Six and a half years from filing to first token, and 57 cents of every dollar never reaches a semiconductor company. CoreWeave passed one gigawatt of active power this spring and runs it at $8.31 billion of annualised revenue, so the scale is real and it is operating today.

The cost is the easy half. The harder question is what happens to it afterwards, and that is where this got interesting.

The physical plant lasts fifteen years and is financed on paper of about that tenor. The silicon that generates the revenue to service that paper has an economic life of three to four years, and loses most of its pricing power inside each turn. Nearly every unusual structure in this market exists to manage that gap.

Pricing the output turned out to matter more than pricing the input. At the wholesale rate CoreWeave’s own disclosures imply, the campus returns 13.4% over fifteen years. Drop fleet utilisation from 90% to 70% and that becomes 1%. Not a crash, not a demand shock. An ordinary year with some idle capacity, against a silicon bill that arrives every four years regardless.

What follows is the whole stack, because the arithmetic is the argument. The worked example below is a quarter of a gigawatt, which is the size real projects actually get built at. Multiply by four wherever you like.

Three clocks

Start with why the physical layer takes so long. NVIDIA ships a new rack platform every twelve months. A building takes 24 to 36 months. Getting power takes four to seven years.

A hall specified for 130 kW racks in 2025 energizes in 2030, into a market shipping 600 kW racks. Most of what looks odd about this sector follows from that: operators overbuilding electrical headroom they cannot populate for years, the premium on already-energized industrial sites, the sudden interest in behind-the-meter gas, and the pricing power sitting with anyone who can shave a year off a schedule.

Hyperscaler capex will land between $600 and $725 billion this year. Dell’Oro has global data center capex crossing a trillion. Nobody thinks money is the constraint. Transformer lead times are past 160 weeks. Switchgear is booked through 2028. Of the 12 GW of U.S. capacity supposedly arriving in 2026, about a third is under construction. The rest is financed, permitted, and stuck.

Which is the useful way in. If capital is not the constraint, then the only way to understand this buildout is to price the things that are.

Aerial or elevated view: steel frame, substation pad, laydown yard in one frame. The laydown yard is the detail worth seeing. It stays for years.

What the $37.6 billion buys

Everything below refers to the same project. I derived the equipment counts rather than asserting them, so you can change an input and rerun it.

Two need a flag. The transformer bank has to be sized on the N-1 case, not the peak. At 313 MVA, losing one unit of a four-by-100 MVA bank leaves 300 MVA and the campus sheds load, so the units go to 120 MVA and N-1 carries 360. And the flow rate assumes 75% of heat leaves through liquid. The other 25% still needs air handling, so liquid-cooled halls still have CRAH units, and retrofit budgets that skip this come in low.

Schedule: 78 months from interconnection filing to first token. The four longest items on the critical path are all electrical. Note the sequencing that forces. The transformer order goes in around month 12, two years before the building it serves is designed in detail, often before a tenant signs. Speculative transformer orders are now normal, and there is a working secondary market in reserved factory slots.

Power

Eight voltage conversions sit between the transmission line and a GPU. Each is a separate industrial supply chain with its own oligopoly and its own way of failing.

The stage that surprises people is the last one. Everything to the left of the rack existed in some form in a 2015 data center. The 800 VDC shelf and the rack-level battery do not have an analogue, and they are there for a specific reason: a training collective makes tens of megawatts appear and disappear in milliseconds, and the distribution system has to absorb that without passing it upstream to the utility.

Land and the queue

Site selection used to weigh latency, tax abatement and fiber. Now it weighs one thing: how fast can this site get power. In practice developers screen first for an existing queue position, or an energized substation with a spare bay. Then transmission within a few miles, on two independent corridors. Then room for gas or batteries as a bridge. Then water rights. Then whether the local union hall can field a thousand electricians for two years.

PJM projects entering service in 2025 averaged over seven years: three-plus to get an interconnection agreement, then about four more waiting to energize. The bottleneck moved downstream. It is no longer the study, it is physical delivery of transmission and equipment.

The base rate is worse than people assume. Of all capacity that entered queues between 2000 and 2019, 13% had reached commercial operation by the end of 2024. Entering a queue is easy. Clearing one is not.

230 kV customer substation serving a data center, Quanta Services

Transformers and switchgear

A large power transformer is bespoke: grain-oriented electrical steel, copper windings, custom bushings, a test bay, and a heavy-haul route to site. You cannot flex that capacity on any useful timescale.

Eaton has guided to solid-state transformer orders in the second half of 2026, shipping late 2027. First real technology relief valve, and it arrives after this capex wave. Watch whether those orders show up as commercial-scale commitments from named hyperscalers or as pilots. The difference tells you whether it is real.

Switchgear is effectively sold out through 2028, with high-voltage breakers at 18 to 24 months. Eaton’s numbers show what that does to a P&L. Q1 2026 revenue was $7.45 billion, up 17%, on a $22.8 billion backlog. But data center orders were up 240% and data center revenue up 50%. When orders grow fourteen times faster than the corporate top line, mix is the whole story. Forgent Power Solutions has a nearly $2 billion revenue backlog, primarily across switchgear and other power infrastructure products, up 157% year over year.

Which raises a question the sector rarely answers directly: why do these lead times turn into margin rather than share loss? Because the layers are concentrated enough that nobody has to compete on price.

The top five suppliers control 62% of North American data center power. In transformers the concentration is higher still, capped by grain-oriented steel and test-bay capacity rather than by capital. Rack integration is the instructive counter-example: Foxconn and Quanta hold 65 to 70% between them and still run single-digit to low-teens gross margins. Concentration without pricing power is just concentration.

UPS, generators, busway

AI training does not draw power smoothly. It draws a synchronised square wave, tens of megawatts stepping on and off in milliseconds as an all-reduce completes across the cluster. That is a power-quality problem, not a capacity problem. Vera Rubin includes 20 times more on-rack energy storage than Blackwell purely to smooth it.

The bigger consequence gets missed. A training cluster restarts from checkpoint, so losing power costs you minutes, not data. Several AI-native operators have concluded that full 2N generator backup on training halls protects against a failure the workload already tolerates, and are building to lower tiers. On this campus that is the difference between about ninety 3.25 MW gensets and almost none. Call it $250 to $350 million and a year of procurement.

Busway is the one nobody was watching. Sightline Climate added it to the long-lead list for the first time in May 2026, which tells you the constraint now covers the entire path from substation to server rather than one point on it. Moving to 800 VDC pushes 150% more power through the same copper and removes some 200 kg of busbar per rack, about 200 tonnes across this campus.

Heat

Every watt into a GPU comes back out as heat. This campus is a 250 MW heater. Water carries about 3,500 times more heat per unit volume than air, so past 50 or 60 kW per rack, air stops being an option.

The water-versus-power trade decides more than it looks like it should. Evaporative cooling on this campus runs on the order of a billion gallons a year. Going fully dry eliminates that and costs about 30 MW of extra electrical load. At a fixed 300 MW service, 30 MW is about 140 Vera Rubin racks you no longer get to install. Water and compute are directly interchangeable, and in most 2026 markets water is the cheaper of the two.

The 45°C specification follows from the same logic. Chillers eat a fifth of facility power. A plant rejecting heat at 45°C instead of 18°C runs free cooling most of the year in most climates. Moving PUE from 1.30 to 1.12 on a 300 MW service frees about 40 MW, or 190 more racks, with no new interconnection and no new transformer. Best return on capital in the building.

Note who has been buying: Eaton acquired Boyd Thermal in March 2026, Schneider owns Motivair. The electrical incumbents bought into thermal because customers want power and cooling as one reference architecture. Vertiv already had that, which is much of why it holds the position it does.

Source: Mitsubishi Electric

Compute

Vera Rubin replaces the cable backplane with a PCB midplane, runs a liquid-cooled busbar at 50 V into the compute trays with DC-DC shelves stepping down from 800 V, and carries 20 times the on-rack energy storage. Kyber in 2027 turns eighteen blades vertical to fit 576 GPUs into one scale-up domain.

Slab-to-slab height, floor loading, pipe diameter and electrical room area are fixed when you pour. When you underwrite a colo operator in 2026, the design density of the newest hall tells you more about terminal value than current occupancy does.

Source: Nvidia

Networking, optics, storage

Three separate networks, and conflating them is the most common analytical error here. Scale-up is NVLink over copper inside the rack, 72 packages at 3.6 TB/s per GPU, under two metres. Scale-out is Ethernet or InfiniBand across the hall at 800G moving to 1.6T, almost entirely optical. Scale-across is coherent DCI between buildings, which exists because single-site power has capped out and a frontier run now has to span campuses.

The constraint underneath all three: an idle GPU is the most expensive object in the building. A $4 million rack stalling on a gradient all-reduce burns capital faster than any network saving recovers, so AI fabrics run near 1:1 subscription and eat far more optics per unit of compute than cloud ever did. This campus takes about 250,000 transceivers and 5,000 km of fibre. Call it $200 million of optics.

One structural oddity. In every previous optical cycle a speed generation ramped, plateaued and got replaced. This time 800G, 1.6T and 3.2T are ramping simultaneously. That holds demand across more of the supply chain and delays the inventory digestion that ended every prior cycle. Whether it survives 2027 and 2028 is the open question for the sector, and I have not seen a convincing answer either way.

Storage gets ignored and causes disproportionate pain, and the driver is checkpointing rather than capacity. A two-trillion-parameter model in bf16 carries about 4 TB of weights, 8 TB of fp32 master weights and 16 TB of Adam optimiser moments. Roughly 28 TB per checkpoint, and the cluster sits idle while it writes. Keeping that stall under ten seconds needs 2.8 TB/s of write bandwidth. That number sets the architecture. It also explains why parallel filesystem vendors keep showing up as gating suppliers on cluster bring-up.

Commissioning

Integrated systems testing runs eight to fourteen weeks and reliably finds control-sequence failures that no individual system test catches. A chiller that fails to restage after a generator transfer. A CDU that alarms during a voltage sag. A building management system fighting the electrical monitoring system. Then GPU burn-in, days of synthetic all-reduce hunting for marginal transceivers and badly seated cold plates.Assume three to nine months between mechanically complete and revenue generating, and discount capacity announcements accordingly. A delayed 60 MW facility costs about $14 million a month in foregone revenue. Operators will pay almost anything to compress that window, and commissioning agents have quietly become a constrained resource of their own.

What the output costs

$9.4 billion and six and a half years spent. An hour of GPU time costs this much to produce.

72,700 packages times 8,760 hours at 90% availability gives 573 million billable GPU-hours a year. Against that: $4.0 billion of IT on a four-year life, $5.4 billion of infrastructure on fifteen, 2,144 GWh at six cents, and O&M at 2% of infrastructure capital. Cash cost is $2.78 per GPU-hour.

Note the split, because it is the argument. Fifty-seven percent of the capital contributes 63 cents an hour. Forty-three percent contributes $1.74. The infrastructure is cheap per unit of output precisely because it lasts, and the silicon is expensive precisely because it does not.

What the output sells for

This is where most models of this business go wrong, including my first pass at it. There is no single price per GPU-hour. There is a decay curve, and it is steep.

GB200 rack-scale capacity currently rents between $10.50 and $27.04 per GPU-hour across tracked providers. H100, two and a half years old, sits at $2.43 to $2.63 on the neocloud tier. B200 and H200 fall in between, in vintage order. That is not noise around a mean. It is a curve.

The implication matters more than the numbers. Revenue per GPU decays at close to the same rate as resale value. The residual on the silicon is not a separate exposure that shows up when you sell the fleet. It is the same exposure, observed twice, and it starts eroding cash flow in year two.

One caveat before the returns math, because everything downstream depends on it. Those are published on-demand rates. A 250 MW campus does not sell on-demand at retail. It leases wholesale to a lab or a hyperscaler, and that realised price is not published. Rather than pick a number and pretend, I have expressed everything below as a percentage of the retail index, and then checked the answer against a company that discloses enough to derive it.

Does it earn the capital back?

Cash cost is not where anyone should stop. The question is whether the campus returns its capital across a full infrastructure life, with silicon replaced every four years and each new generation entering at the top of the decay curve.

At a 55% wholesale realisation, which implies a lifetime average around $4.04 per GPU-hour, the campus returns 13.4%. Breakeven against an 8% unlevered hurdle is 49% of the retail index, or about $3.62 average. Above that it works. Below it, it does not, and at 35% the project destroys capital outright.

But price is not the variable that decides this. Utilisation is. Moving from 90% to 70% fleet utilisation takes the IRR from 13.4% to 1.0%. Price would have to fall by a fifth to do equivalent damage. The reason is the treadmill: $4 billion of silicon has to be bought every four years whether the fleet is busy or not, and a half-idle fleet still pays for it in full.

Almost nobody discloses utilisation. If you take one operational metric away from this piece, make it that one.

At a 55% wholesale realisation, which implies a lifetime average around $4.04 per GPU-hour, the campus returns 13.4%. Breakeven against an 8% unlevered hurdle is 49% of the retail index, or about $3.62 average. Above that it works. Below it, it does not, and at 35% the project destroys capital outright.

But price is not the variable that decides this. Utilisation is. Moving from 90% to 70% fleet utilisation takes the IRR from 13.4% to 1.0%. Price would have to fall by a fifth to do equivalent damage. The reason is the treadmill: $4 billion of silicon has to be bought every four years whether the fleet is busy or not, and a half-idle fleet still pays for it in full.

Almost nobody discloses utilisation. If you take one operational metric away from this piece, make it that one.

Checking the model against something real

A 55% realisation is the load-bearing assumption in everything above, and I do not like load-bearing assumptions I cannot test. CoreWeave, as it happens, publishes both halves of the ratio.

CoreWeave reported $2.08 billion of revenue in Q1 2026, up 112%, and crossed one gigawatt of active power during the quarter. Annualised against that exit figure, revenue runs at $8.31 million per megawatt of active power per year.

Treat that as a floor rather than a midpoint. They crossed a gigawatt during the quarter, so the revenue was earned on a lower average base and the true figure per active megawatt is higher. Full-year guidance of $12 to $13 billion, against a ramp from 1.0 to more than 1.7 gigawatts, implies about $9.3 million.

My model produces $8.05 million per megawatt at a 55% realisation and $9.51 million at 65%. So the disclosure brackets a realisation of 55 to 65% of the retail index, which is an IRR range of 13 to 22% rather than the single 13.4% figure. I have kept 55% as the working assumption throughout because it is the conservative end, and because the honest position is a range.

Two things that check does not do, and I would rather say them than have someone else. CoreWeave leases a large share of its capacity rather than owning it, so its income statement carries rent where my model carries depreciation. Its 56% adjusted EBITDA margin is therefore not comparable to anything here. And its fleet spans Ampere through Blackwell, so the figure is blended across vintages. That makes it the right comparison for a lifetime average and the wrong one for year one.

What it does do is move the weakest input from assumption to a bounded range anchored on a filing. If you have better wholesale data, you can see exactly which number to replace and what it does to the answer.

One more number from the same filing. CoreWeave guides to $31 to $35 billion of capex this year against $12 to $13 billion of revenue, having raised the capex range mid-year. Capex is running at more than two and a half times revenue. That is what the treadmill looks like on a real cash flow statement.

So which depreciation schedule is right?

I said earlier that hyperscalers book five to six years and AI-natives book three, and that both cannot be right. That was too glib. Having worked the decay curve, I think both are right, and the reason is more useful than the dispute.

H100 SXM5 cards sold for around $40,000 in late 2023. They move for $6,000 to $15,000 now. Roughly 70% in about thirty months, and the decline is back-loaded: value holds through the first two years, then falls away as the next generation arrives in volume. A five-year straight line implies about 20% annual decline. The observed curve is nothing like that.

The mechanism is not mysterious. Inference on an H100 costs eleven times what it costs on a B300. Once the newer part ships in volume, the older one cannot be priced competitively for the workload that would otherwise keep it busy. The hardware still works. The economics stop.

This is the part I got wrong the first time. That argument is decisive for an operator whose revenue comes from the rental market. It is much weaker for one with internal demand.

A hyperscaler at year four cascades the fleet to search ranking, recommendations, ads inference and internal serving. None of that needs frontier performance, the hardware stays busy, and a five to six year life is defensible because there is a workload backstop that does not depend on rental pricing at all. A neocloud has no such backstop. Its year-four fleet competes in the open market against the H100 floor of $2.40 to $2.60.

The schedules are not wrong. They are being borrowed. Operators without internal workloads are applying useful-life assumptions that were derived by companies who have them, and lenders are underwriting against the borrowed number. That is a narrower claim than I started with and I think it is the correct one.

How it is financed

Which brings us back to the top. If the revenue-generating asset has a three to four year life, what is the debt behind it?

The scale here is easy to miss. Total data center debt issuance nearly doubled to $182 billion. Private credit lending to AI-related companies went from near zero to over $200 billion in a few years, and Morgan Stanley projects another $800 billion over the next two. JPMorgan expects $30 to $40 billion of annual data center securitisation across 2026 and 2027, which would be 7 to 10% of combined CMBS and ABS issuance.

Meta’s Hyperion is the template. A special purpose vehicle called Beignet Investor, arranged by Morgan Stanley, raised $27 billion of A-plus rated debt anchored by PIMCO and BlackRock, plus equity from Blue Owl. Meta keeps operational control, leases the campus back, and converts what would have been capex into predictable opex. The structure is elegant and it is being copied.

Where the money goes

Coverage of AI capex collapses almost entirely to accelerators. Accelerators are 43% of the bill. The other 57% goes to a supply base with multi-year backlogs and, for the first time in decades, real pricing power.

Contractor backlogs are the highest-quality real-time data in the sector, because backlog is signed contract booked 12 to 24 months before revenue. Comfort Systems ended June with $14.06 billion against $8.12 billion a year earlier. Same-store went from $8.12 to $13.70 billion. Q2 revenue up 50%, EPS up 92%, operating cash flow over a billion in a single quarter. Three straight quarters of same-store backlog growth above a billion, with duration running into 2027 and 2028.

On the multiples: Vertiv at 44 times forward is the purest expression of the thesis and leaves no room for a guidance miss. Eaton at 30 times looks slower only because data center is a minority of a very large book, so the 240% order growth is the more useful number. Comfort Systems and EMCOR trade on contractor multiples against utility-like visibility, and that is the clearest dislocation in the group. It is also the most exposed if conversion slips.

The portfolio implication is the part I keep coming back to. The suppliers to the physical layer have this pointed the right way round. They sell into a fifteen-year asset, they book backlog that converts over 12 to 24 months, and they carry no residual risk on the silicon. The operators and the neoclouds carry all of it. Owning the picks and shovels is not just a way to get AI exposure without picking a model winner. It is a way to be long the buildout while short the specific thing that makes the buildout fragile.

How this breaks

The efficiency shock is the one I actually worry about, not because it is likely on any particular timeline but because it is the only scenario where demand for megawatts falls while demand for AI rises. Nobody in this supply chain is positioned for that. Everything else on that list is a timing risk.

A demand air-pocket in the next four quarters is the one I take least seriously, and the backlog data is why. You cannot quietly cancel $14 billion of signed mechanical contracts. The contractors would report it before the hyperscalers guided to it.

The scenario the table understates is slower and duller. No collapse, just utilisation drifting into the low 70s while depreciation gets marked toward the secondary market. On the model above that combination takes the IRR from 13.4% to about 1% without a single dramatic headline, and it lands on the operators carrying SPV debt rather than on the suppliers.

What to watch

Everything in this piece reduces to a small number of things you can actually watch.

What this actually is

This gets framed as a software story funded by capital markets. It is closer to a heavy industrial construction story with a chip company at one end and a utility at the other, running through a supply chain that spent twenty years optimising for a demand curve that no longer exists.

Fifty-seven cents of every dollar in that $9.4 billion never reaches a semiconductor company. It goes to transformers with three-year lead times, switchgear sold out through 2028, chillers and CDUs and two hundred tonnes of copper busbar, and about 2,500 people spending two years on a site in Ohio or Louisiana turning capital into megawatts.

A gigawatt costs $37.6 billion, takes six and a half years, and 57 cents of every dollar of it is transformers, switchgear, chillers, copper and labour. That physical plant is the durable half. It costs 63 cents an hour to run, it lasts fifteen years, and nobody can build it quickly. The problem is that it is financed on the strength of revenue from the other half, which turns over five times before the paper matures and loses most of its pricing power inside each turn.

At the realisation rate CoreWeave’s own disclosure implies, the campus returns 13.4% and everyone is fine. At 70% utilisation it returns 1%. The distance between those two outcomes is not a market crash. It is an ordinary year with some idle capacity.

The buildings will be fine. Watch who is holding the silicon.

The short version

Ethernet didn’t outrun InfiniBand, it absorbed it. RDMA in 2010, adaptive routing in 2023, in-network collectives in 2025, each clone arriving faster than the last, until Ultra Ethernet started shipping capabilities InfiniBand never had. The share flip is done, from over 80 percent InfiniBand in late 2023 to roughly two-thirds Ethernet ever since 2025. NVIDIA’s answer was to sell the winner, and its networking business now runs near $60 billion annualized. Now the war is climbing into the rack, where AMD’s UALink-over-Ethernet systems launched this week, and the durable value is pooling at the endpoint, in the NIC, the rack-scale fabric, and a collectives software layer that nobody owns yet.

What the ending of a standards war teaches

Every large AI cluster is really two computers. The first is the one everyone talks about, the racks of GPUs. The second one decides whether the first is actually earning its keep. That’s the network stitching tens of thousands of GPUs into a single machine. When the second computer fails, the first one stops earning. At frontier scale something in the cluster breaks every few hours, and because training is synchronous, a single flapping optic or misbehaving NIC doesn’t degrade one server, it stalls all of them. On a 100,000-GPU cluster, roughly $4 billion of silicon renting for about $300,000 an hour at market GPU rates, a failure that takes thirty minutes to detect and checkpoint-restart burns $150,000 of compute. And a failure that never quite resolves, a marginal link retraining every few minutes, becomes a tax on every step of every epoch of every run.

That failure math is why the fabric stopped being a plumbing decision and became a boardroom one, and why the fight over it, between InfiniBand, the purpose-built interconnect NVIDIA acquired with Mellanox, and Ethernet, the fifty-year-old open standard, turned into the biggest open-versus-proprietary fight in infrastructure since Linux took the server. Here’s the thing though. That war has effectively been decided. NVIDIA’s own networking segment, the house that InfiniBand built, now runs at nearly $60 billion annualized and its growth is led by Ethernet products. So this piece is not really about who won. It’s about the mechanism by which the winner won, because that mechanism is about to run again one tier up the stack, and this time you can watch it with your eyes open.

One chart carries the whole argument. For twenty-five years, every technical advantage InfiniBand shipped eventually showed up on Ethernet. Track how long each one stayed exclusive and you get the war’s entire logic in five rows.

Three lessons hang off that chart, and they organize everything below. First, in networking, a proprietary advantage is a rental, not a moat, and the rent is set by how hard the feature is to standardize. Second, the clones arrive faster each cycle because the buyers switched sides, and the Ultra Ethernet Consortium is nothing more than the customers collectively funding the cloning of their supplier’s moat. Third, the moment the copier starts originating, the war is over and the front moves. Ultra Ethernet’s spray-native transport is that moment. The new front, the rack fabric and the endpoint, opened this week.

Three moats, three half-lives

InfiniBand held three real technical moats. Each one fell by a different mechanism, and each mechanism teaches you something about the next war, so it’s worth being precise about what actually happened rather than reciting the acronyms.

The first moat was latency, and it was never made of wire. A kernel-mediated TCP round trip costs tens of microseconds in context switches and buffer copies. InfiniBand’s RDMA did the same work in about one, a 40x gap that mattered enormously when a collective operation finishes only when the last message lands. But the moat was software, kernel bypass and a hardware transport, and software travels. When RoCE ported InfiniBand’s own transport onto Ethernet in 2010, the 40x premium collapsed to roughly 2x, and at the megabyte message sizes that dominate gradient exchange, 2x on first-packet latency is noise against bandwidth. The lesson is that physics moats are rare and most latency moats are actually code, which means they are one port away from evaporating.

The second moat was congestion behavior, and its fall is the one that matters most for what comes next. AI traffic is a small number of enormous, identical-looking flows, and classic Ethernet load balancing hashes whole flows onto single paths, so two elephants collide on one spine link while the rest sit idle. Meta’s published RoCE work documents the failure mode at production scale, and NVIDIA’s marketing pegs commodity ECMP at roughly 60 percent delivered bandwidth, a vendor number, but one whose direction nobody who runs these fabrics disputes. InfiniBand avoided this with central scheduling and credits. Ethernet’s first answer, PFC plus DCQCN, worked but ran on operational pain, pause storms and victim flows and heroic tuning. The answer that stuck is packet spraying, splitting every transfer across all paths and reordering at the destination, which is what Spectrum-X, Ultra Ethernet, AWS’s SRD and now MRC all do. Notice where the reordering happens. Not in the switch. In the NIC. The durable fix for Ethernet’s biggest weakness was to move intelligence out of the network core and into the endpoint, and that relocation is the hinge of this entire piece.

The third moat was computing inside the network. SHARP lets InfiniBand switches sum gradients as they flow through, roughly doubling effective allreduce bandwidth, and it survived nine years without an open equivalent because it requires co-design across switch silicon, NIC and the collectives library. That is the longest half-life on the chart, and it explains NVIDIA’s entire subsequent strategy. When your co-designed features are the only ones that hold, you push co-design deeper, which is exactly what the NVL72 rack is. The lesson reads both ways. Co-design buys the longest exclusivity, and even it expires, since UEC 1.0 specified In-Network Collectives and Broadcom’s Tomahawk Ultra shipped collective offload in 2025.

The endgame came fast once the buyers organized. The Ultra Ethernet Consortium, founded in 2023 under the Linux Foundation by AMD, Arista, Broadcom, Cisco, HPE, Intel, Meta and Microsoft among others, shipped its 1.0 spec in June 2025, and it doesn’t imitate InfiniBand so much as leapfrog it. UET sprays packets across every viable path, delivers out of order while completing in order, starts transmitting before handshakes finish, and trims congested packets into a fast-lane loss signal instead of dropping them silently, targeting fabrics of a million endpoints. Then in May 2026 the model labs took the pen themselves. MRC, from OpenAI, Microsoft, Broadcom, AMD and NVIDIA, grafts the same ideas onto existing RoCE across eight parallel data planes, and its geometry is the real tell. It supports 131,072 endpoints in two switching tiers instead of three, three hops instead of up to seven, and double the compute for roughly 20 percent more switches. Meanwhile the proof of practice had already landed, since Meta built twin 24,576-GPU clusters as an explicit A/B test, Quantum-2 InfiniBand against RoCE on Arista, and trained Llama 3 on the Ethernet one.

The receipts are in the market data. In late 2023 InfiniBand carried over 80 percent of AI back-end switch spend. Ethernet crossed half during 2024, took more than two-thirds for full-year 2025, and held that share through 1Q 2026 even as InfiniBand sales more than tripled year on year on the Blackwell Ultra ramp. Both fabrics are growing absolutely. Only one is growing structurally, and the vendor behavior underneath tells you which, since Celestica retook the top Ethernet spot, Cisco posted the quarter’s biggest share gains, and 1.6T switches are sampling for an H2 ramp, the kind of multi-vendor churn InfiniBand’s single-supplier structure cannot produce.

NVIDIA’s counter was to sell the winner

Watch what the incumbent did when it saw the chart above forming, because it’s the cleanest execution of the losing-a-standards-war playbook on record. In 2023, with Meta already on RoCE and AWS never having adopted InfiniBand at all, NVIDIA launched Spectrum-X, its own AI Ethernet, pairing the Spectrum-4 switch with a mandatory BlueField SuperNIC and doing per-packet adaptive routing with reordering at the endpoint. InfiniBand techniques in an Ethernet frame. It claims about 95 percent effective bandwidth against roughly 60 for commodity ECMP Ethernet, vendor numbers worth your skepticism, but the deployments are not marketing. xAI’s Colossus stood up 100,000 H100s in 122 days on Spectrum-X and has since grown past a reported half-million GPUs, still on Ethernet. Within two years Spectrum-X passed a $10 billion annualized run-rate and IDC ranked NVIDIA the number one datacenter Ethernet switch vendor by revenue, ahead of Cisco and Arista, in a market running above a $60 billion annualized pace.

The strategic content of Spectrum-X is not the throughput claim. It’s the attach. You can leave InfiniBand and NVIDIA still sells you the switch, the software, and above all the SuperNIC, because the performance features only light up end to end on NVIDIA endpoints. The skeptic’s line is that Spectrum-X is open the way a hotel minibar is convenient, and the skeptics are right, which is the point. NVIDIA read where the intelligence was moving and made sure that even its concession to open standards kept the endpoint proprietary. When the same company then co-authored MRC and opened NVLink Fusion to partners like Marvell, the pattern became unmistakable. Every NVIDIA response to openness protects the same square on the board, the endpoint.

The war doesn’t end, it moves up a tier

Here’s the map that explains why nobody gets to relax. An AI datacenter runs three networks, and they differ by roughly 40x in bandwidth per GPU at each boundary. Inside the rack, the scale-up domain moves 1.8 TB/s per GPU over copper NVLink and the rack behaves like one accelerator. One tier out, the scale-out back-end gives each GPU a 400 or 800G port into the Clos fabric, and that middle band is where the entire war just described was fought. The front-end tier below was always Ethernet and never contested.

Whoever owns a tier taxes everything that crosses it. Every fabric vendor is now trying to move up one band.

The second front opened on schedule, and this week it went live. AMD launched Helios on July 20 with Microsoft committing to deploy it at scale on Azure, a 72-GPU MI455X rack whose scale-up fabric tunnels UALink over Ethernet on switch silicon co-designed by Broadcom and HPE’s Juniper, with Pensando Vulcano UEC NICs handling scale-out and mass production ramping toward Q2 2027. Run the absorption chart’s logic against it. NVLink’s scale-up exclusivity dates to 2014-era SLI lineage but its modern form to 2016, the open clone arrived as a spec in April 2025 and as shipping hardware in July 2026, and NVIDIA pre-emptively opened NVLink Fusion to partners rather than wait to be cloned. The half-lives aren’t just shrinking. The incumbent now starts the absorption itself, which tells you it believes the pattern too.

The tier above the rack opened at the same time. Frontier training now spans buildings and metros, so NVIDIA shipped Spectrum-XGS for what it calls scale-across, with CoreWeave first in line, and Broadcom built Jericho4 for the same job. The corroboration that matters comes from Arista, which expects scale-across to contribute at least a third of its $3.5 billion 2026 AI target after describing the segment as virtually nonexistent a year ago. Inference pushes in the same direction, since disaggregated prefill and decode keep the heaviest traffic inside the rack and turn the middle tier into KV-cache plumbing that plain Ethernet handles fine. Both ends of the stack are pulling value away from the tier InfiniBand just spent three years defending.

Follow the money to the endpoint

Put every public company selling into this war on the same annualized basis and the growth ranking is itself an argument. NVIDIA’s networking segment runs near $60 billion annualized, up 199 percent, growing faster than its compute business. Broadcom’s AI semiconductors run at roughly $43 billion annualized, up 143 percent, with guidance implying a $64 billion pace this quarter and a backlog above $30 billion. Credo, which sells reliability-first copper cables into hyperscale racks, tripled fiscal 2026 revenue to $1.3 billion. Arista grew 35 percent and guided 2026 to $11.5 billion. Cisco’s datacenter switching, the pre-AI networking economy, grew 9. Sum the disclosed pieces, NVIDIA’s networking line, the roughly 40 percent of Broadcom’s AI revenue that is networking, Arista’s AI business and Credo, and AI networking grosses roughly $80 billion annualized before counting optics, cables, private vendors or the white-box builders. Two honest deductions apply, since the fiscal calendars don’t align perfectly and some of Broadcom’s switch silicon ships inside Arista and white-box systems and gets counted twice, so call the clean number $70 billion and change, still well over double what the entire datacenter switch industry grossed in a year as recently as three years ago. A component category became an industry in thirty months.

Now look at composition, because two details in those reports carry the thesis. Broadcom says networking is about 40 percent of its AI revenue and guides that mix toward 30 as custom accelerators ramp, which means the switch is becoming the smaller half of the AI story at the merchant switch champion itself. And the arithmetic of MRC makes the same point at the fabric level. When an MRC-style network doubles its GPUs, NICs double with them, one per accelerator, while the flattened two-tier topology needs only about 20 percent more switches. The endpoint scales with compute. The core deliberately doesn’t. That’s not a side effect, it’s the design goal of every post-2023 fabric, and it redraws where the revenue pools.

Size the two prizes and the migration stops being abstract. Call accelerator shipments 6 to 8 million units a year by 2027, across NVIDIA, AMD and the custom XPU programs, with one AI NIC apiece at $1,500 to $2,500 as 800G becomes the floor. Those are my assumptions, not a vendor’s, and they put the AI NIC pool at $10 to $20 billion a year and compounding linearly with compute, against a back-end switch market that Dell’Oro’s $100 billion five-year forecast averages to roughly $20 billion a year growing sublinearly by design. The endpoint is on course to match the core in dollars before the decade ends, and that’s before scale-up endpoint silicon, which Helios just turned into a merchant category.

The same migration is visible in the physical layer. A scale-out fabric consumes up to six optical transceivers per GPU across its tiers, Ethernet optics grew about 70 percent in 2025 on top of a doubling in 2024, and LightCounting sees a path to $100 billion a year in AI-cluster optics by 2030. Reliability, not just bandwidth, is what’s being priced, since at one interruption every few hours a flapping transceiver is a six-figure event. That’s why Credo can triple revenue selling copper cables whose pitch is a claimed thousandfold reliability edge over pluggable optics, and why NVIDIA is moving optics into the switch package entirely with co-packaged Quantum-X and Spectrum-X Photonics parts arriving through 2026, claiming 3.5x better power per bit. Everyone in the physical layer is selling the same two things now, joules and uptime.

Which brings the argument to its destination, the M&A record, because the buyers have been marking this thesis to market for fifteen years.

Every layer of that table has consolidated except one. The collectives software sitting above the NIC, the layer NCCL occupies, has no independent owner, and understanding why it’s empty tells you why it won’t stay that way. Until roughly last year there was nothing to be neutral about. Fleets were homogeneous, the fabric vendor shipped the software as a control point, NCCL from NVIDIA, RCCL as AMD’s port, and the only buyers rich enough to fund an alternative built private ones instead, NCCLX at Meta, MSCCL at Microsoft. Great abstraction layers only emerge when the hardware underneath fragments. VMware needed commodity server sprawl, Terraform needed multicloud. AI fabrics crossed that line in the last twelve months, with UEC NICs shipping from three vendors, UALink racks arriving alongside NVL72s, and neoclouds running InfiniBand and Ethernet estates side by side. An operator buying Helios and Vera Rubin systems in the same year now owns a fabric zoo with no common software plane. That is a company waiting to exist. Sizing it honestly requires an analogy rather than a bottom-up, because the category doesn’t exist yet. Infrastructure software that manages a hardware layer has historically captured low single digits of the spend beneath it, Datadog runs at roughly one to two percent of the cloud bills it watches, VMware at its peak took closer to ten percent of the server market it abstracted. Two to four percent of software attach on a $70 billion fabric complex is a $1.5 to $3 billion annual revenue pool by the decade’s end, concentrated in whoever wins the neocloud and enterprise tier first, and the founder pool to watch is the short list of people who have actually shipped NCCL, NCCLX or MSCCL.

The layer comes with a data business attached, which is what makes it venture-scale rather than a feature. A frontier cluster takes an interruption every few hours, Meta logged 466 across 54 days of Llama 3 training, and attributing a stalled step to a flapping link, a mistuned rail or a failing NIC is still done with vendor tools that stop at each vendor’s border. The collectives library is the one component that observes every transfer on every fabric in the job. Whoever owns it owns stall attribution, and in a market where an hour of cluster time costs $300,000, stall attribution isn’t a dashboard, it’s a claims adjuster for the most expensive machines ever built.

Now stress-test that thesis the way the steelman below stress-tests InfiniBand, because it can fail three ways. Hyperscalers might in-house the layer, the way they in-housed the NIC itself, since Annapurna became EFA, Google built Falcon, and NCCLX and MSCCL already exist, which would shrink the independent opportunity to neoclouds and enterprises. NVIDIA might integrate it away, folding enough fabric-awareness into NCCL and enough openness into NVLink Fusion that neutrality stops being worth paying for. Or the heterogeneity could prove temporary, since if Rubin-era racks simply win the next two years, fleets re-homogenize and the abstraction moment passes the way OpenStack’s did, and silicon cycles run longer than venture timelines either way. My honest read is that this layer gets built three times, once in-house, once by NVIDIA, and once independently, and the independent version matters only if open endpoints ship at volume on schedule. Which is exactly what the watchlist is for.

The steelman and the scoreboard

The bull case for InfiniBand from here rests on four legs, and it’s a real case. Quantum ships in lockstep with each GPU generation, which is why its sales tripled on the Blackwell Ultra ramp two years into Ethernet’s supposed rout. In-network collectives are specified in Ultra Ethernet but proven only on Quantum, and frontier training pays for that maturity gap today, not in 2027. The next million buyers are enterprises and sovereigns without elite network teams, a population that buys reference architectures that work out of the box, which is how CoreWeave built a business on Quantum fabrics. And co-packaged optics lands on Quantum first in NVIDIA’s own roadmap. The reference-architecture demand is measurable, not anecdotal, since of the 45 systems new to the November 2025 Top500, 31 chose InfiniBand and only 4 chose RoCE. Declining to a third of a market that quintuples is still a growth business. The share data says the war is decided, not that the loser disappears.

A thesis you can’t falsify is a mood. Six markers over the next eighteen months, each phrased so it can prove me wrong.

InfiniBand won the opening battle because it was the only fabric that treated the network as part of the computer. Ethernet won the war for the oldest reason in infrastructure. Given enough volume, open and good-enough runs down proprietary and perfect, and then stops being merely good enough. The next war, for the rack and the endpoint, started this week with the same armies in the same formation. This time, at least, we have the pattern.

Sources & further reading

Market data · Dell’Oro, 1Q 2026 AI scale-out report (Jun 2026) · Dell’Oro, “Ethernet is Winning the War Against InfiniBand” (Jul 2025) · Dell’Oro, 2025 full-year · Dell’Oro, $100B five-year forecast · IDC on NVIDIA taking #1 in DC Ethernet switching
Public financials · NVIDIA Q1 FY2027 results (May 2026) · Broadcom Q2 FY2026, $10.8B AI revenue (Jun 2026) · Arista Q1 2026 results (May 2026) · Credo FY2026 results, revenue triples (Jun 2026)
Standards and transports · UEC 1.0 announcement · VIAVI on UE 1.0 internals · The Next Platform on MRC (May 2026) · NVIDIA on RDMA and RoCE history · SHARP in-network computing
Deployments · Meta’s GenAI infrastructure · Meta SIGCOMM ‘24, RDMA over Ethernet at scale · Llama 3’s 466 training interruptions · Spectrum-X at xAI Colossus · Colossus at 2 GW and 555K GPUs · Microsoft Fairwater
Endpoints, optics and scale-up · CNBC on the AMD Helios launch (Jul 20, 2026) · Helios UALink-over-Ethernet details · Broadcom Thor Ultra · AMD Pollara · NVIDIA’s ~$900M Enfabrica deal · LightCounting on $100B AI optics · NVIDIA co-packaged optics · Spectrum-XGS · June 2026 Top500

• The frontier is no longer proprietary. Kimi K3 scores 57.1 on the Artificial Analysis Intelligence Index, #3 overall behind Claude Fable 5 (59.9) and GPT-5.6 Sol (58.9). The weights ship July 27. The last release cycle closed 13 points of gap in three months; 2.8 remain.

• Margin gravity: closed labs run API gross margins estimated above 80%; open-model serving clears at ~40–45%. On a ~$250B 2030 inference market, that 35-point spread is ~$90B a year of margin migrating from model IP to compute and serving.

• Bearish for labs: US labs’ OpenRouter token share fell ~70% → ~30% in twelve months. The moats that remain — product harnesses and internal checkpoints — are real, but nothing in the usage data prices them yet.

• The caveat: K3 is a token wastrel: $0.94 per benchmark task vs. $0.55 for GPT-5.6 Terra and $0.31 for Grok 4.5. The true Sputnik is a token-efficient open frontier model. Not yet.

• Bullish for compute, memory and serving: CoreWeave backlog $99.4B, Oracle RPO $638B, 2026 capex ~$725B — and export controls route open-model demand onto Western GPUs. Fireworks just repriced from $4B to $17.5B in seven months.

• Five falsifiable calls at the end, starting with: K3 serves below $10 per million output tokens within 30 days of the weights dropping.

What Actually Shipped

The frontier became a file

imi K3 is a 2.8-trillion-parameter mixture-of-experts model: 896 experts, 16 active per token, roughly 50B active parameters, a 1M-token context window, weights trained natively in 4-bit precision. On Artificial Analysis’ independent index it scores 57.1, third overall behind Claude Fable 5 (59.9) and GPT-5.6 Sol (58.9). At launch it took #1 on the Frontend Code Arena at 1,679 Elo, above Fable 5’s 1,631 — the first Chinese model to top a major coding leaderboard.

The weights go public on July 27. That sentence is the entire story, because it converts the frontier from a service into a file. A service has pricing power. A file has distribution.

The producer economics explain why this keeps happening. Moonshot’s K2 Thinking was reported by CNBC to have cost roughly $4.6M to train — the company calls the figure unofficial — against Stanford-estimated compute costs of $78M for GPT-4 and $191M for Gemini Ultra. The velocity is the tell: six frontier-class releases in twelve months, ARR doubled from ~$100M to ~$200M in roughly six weeks this spring, valuation up from $2.5B in early 2024 to $20B in May with a $30B round reportedly forming ahead of a Hong Kong IPO.

K3’s price is its own signal. At $3 per million input tokens and $15 per million output, it costs more than 3x its predecessor. The era of super-cheap Chinese AI is ending. The era of free frontier weights is not — Moonshot is telling you open intelligence now clears at closed-tier prices.

Margin Gravity

Sizing the pool that moves

The cleanest frame for what K3 means comes from Gavin Baker. A world with two or three dominant frontier labs earning 90%+ inference margins is a world where those labs become monopsonies: the price-setting buyers of power, data centers, and semiconductors. They vertically integrate into every layer beneath them and subsume the application layer above. Awesome for those labs. Net negative for everyone else in the economy.

“A world where there are only 2–3 dominant frontier labs with 90% inference margins is net negative for every other layer while being awesome for those 2–3 labs… Anything that lowers margins and increases competition at the model layer is good for every other AI layer: power, semiconductors, hyperscalers, neoclouds and yes even software.”

Gavin Baker (@GavinSBaker), on Kimi K3 as “an important inflection point for AI,” X, July 2026

Now put numbers on it. The 90% figure is not hypothetical: SemiAnalysis estimates Anthropic’s API gross margins above 80%. When the same workload runs on an open model through an inference platform, the model toll disappears and the serving layer keeps an estimated 40–45% (Tom Tunguz pegs Together at ~45%, Groq at ~40%). The compute underneath is identical. Baker’s point, exactly: an open model requires the same silicon as a closed model of the same size and architecture. Only the toll changes hands.

Scale that spread. Forecasters put AI inference around $250B a year by 2030. On that base, every point of model-layer gross margin is $2.5B a year. The spread between an 80%-margin closed API and a 45%-margin open serving stack is 35 points: roughly $90B a year of margin that lands in compute, serving, and silicon instead of model IP — if workloads migrate. That is the pool K3 puts in play. Not revenue. Margin.

There are two roads to this repricing, and both are being paved at once. Open weights at the frontier is one: that is Moonshot. The other is vertically integrated players who don’t care where the margin dollars land: Grok 4.5 pricing output at $6 per million tokens, Meta’s Muse Spark 1.1 at $4.25, Gemini running on Google’s own TPUs. This is also why Jensen Huang champions open source, and it is not sentiment. A diffuse model layer dissolves the monopsony before it forms — and K3 is the first time the open road reached the frontier’s doorstep.

Bearish: The Labs

The frontier premium now depreciates in months

The closed labs sell exactly one thing at those margins: the gap between their best model and the best model you can download. That gap used to depreciate over years. K2.6 to K3 closed 13 index points in three months; 2.8 remain. The premium is still real. Its half-life is now measured in release cycles.

The market has been voting for a year. US labs’ share of tokens routed through OpenRouter fell from roughly 70% in June 2025 to ~30% in June 2026. DeepSeek alone is now the largest model family on the platform at 16.3% of volume.

Set that against the leader’s cost structure. OpenAI booked $13.1B of 2025 revenue against a reported $38.5B net loss, and carries $1.4T in compute commitments. That is a business that must defend premium pricing for a decade, at the exact moment premium pricing became optional for a growing share of workloads.

The labs’ own capital allocation concedes the point. OpenAI is designing 10GW of custom accelerators with Broadcom and bought 6GW of AMD supply. Anthropic contracted up to a million Google TPUs and committed $30B to Azure. Vertical integration is what you do when you expect the model layer’s margin to stop paying for the buildout.

Baker calls K3 only potentially negative for OpenAI and Anthropic. He grants them two outs, and both deserve to be taken seriously.

First, the product may now matter more than the model. Claude and ChatGPT are harnesses: memory, tooling, distribution, enterprise trust. Anthropic’s reported $47B run-rate is a workflow story sitting on top of models, not token resale — and it holds 61% of gateway spend on 32% of gateway tokens, which is what a working moat looks like. Second, the labs may hold internal checkpoints well ahead of anything public, compounding through recursive self-improvement. A few months of lead there could become permanent.

Both are real. Neither shows up in the direction of the usage data.

The dollars-versus-tokens gap is how you watch the race. Menlo Ventures put enterprise workloads at 89% closed at the end of 2025. Vercel’s gateway shows open-weight models carrying 29% of tokens and under 4% of spend, with token share nearly tripling in two months. Tokens lead. Dollars follow, unless the harness moat holds.

The Token-Efficiency Caveat

K3 is a token wastrel, and that cuts both ways

Intelligence per dollar is the product of two terms: price per token, and intelligence per token. K3 wins the first and loses the second.

It burned 132M output tokens running Artificial Analysis’ evaluation suite, more than double the ~63M median, because it ships with always-on maximum reasoning and no effort controls yet. The result is $0.94 per benchmark task: cheaper than GPT-5.6 Sol ($1.04), half of Claude Opus 4.8 ($1.80), but 71% above GPT-5.6 Terra ($0.55) and 3x Grok 4.5 ($0.31). The rough edges don’t stop there — K3’s hallucination rate on AA’s Omniscience test worsened to 51%, from K2.6’s 39%.

So the discount is real at the top of the closed stack and absent against its efficient tier. Running AA’s full index cost ~$2,700 on K3 versus ~$6,200 on Claude Fable 5, the most expensive model AA has ever benchmarked — but a closed lab routing to a token-efficient model at Terra’s price still undercuts the open alternative on delivered work. This is why Baker calls K3 an inflection point rather than a Sputnik moment. The true Sputnik is an open frontier model that is also token-efficient. It has not happened yet.

But inefficiency cuts both ways. Every wasted token is billed compute. A token-inefficient open model is a subsidy from Moonshot’s margin to whoever owns the GPUs it runs on.

Bullish: The Compute Layer

Free models are demand insurance for GPUs

A free frontier model destroys margin in exactly one place, the model API, and creates load everywhere beneath it. K3’s weights are ~1.4TB even at 4-bit precision. Moonshot recommends supernodes of 64 or more accelerators to serve them. Nobody downloads that onto a laptop.

This is Jevons operating at infrastructure scale. As the price of intelligence collapses, consumption explodes, and every unit of it clears through someone’s data center — including K3’s doubled token burn. Google now processes 3.2 quadrillion tokens a month, up 7x in a year. Inference has gone from a third of AI compute in 2023 to a projected two-thirds in 2026.

The operators are printing the evidence. CoreWeave grew Q1 revenue 112% to $2.1B with a $99.4B backlog. Nebius grew 684% and signed $17.4B from Microsoft and $27B from Meta. Oracle’s RPO hit $638B, up 363% year over year. Hyperscaler capex guidance for 2026 stands near $725B, up ~77%, against McKinsey’s $6.7–7T buildout estimate through 2030.

Memory is the purest expression of the trade. K3 activates less than 2% of its 2.8 trillion parameters per token, but every one of its 896 experts must sit resident in memory: 1.4TB before it serves a single request. Sparse models economize on FLOPs and gorge on capacity. Long context makes it worse — a 1M-token window is KV cache, and KV cache is memory. Verbosity makes it worse still: K3’s doubled token burn is decode-heavy work, and decode is bound by memory bandwidth, not compute. Open, sparse, long-context, token-hungry models are precisely the workload HBM was built for. The market is already pricing the squeeze: Bank of America puts the HBM market at $54.6B in 2026, up 58%, with DRAM revenue up 51% and total memory past $440B — and Microsoft attributed roughly $25B of its capex guidance to memory and component inflation alone. Margin gravity says the dollars pool at the scarcest physical constraint. In 2026 that constraint is not FLOPs. It is memory.

The geography is the under-priced part. Export controls mean Moonshot cannot serve global demand from Chinese silicon; it serves K3 today at a throttled ~26–28 tokens per second. Releasing the weights exports the serving demand to the only infrastructure that can absorb it: Western GPUs. The Sputnik analogy gets the consequences backwards. Sputnik was a rival’s closed capability. K3 is a demand shock delivered directly to American data centers.

Bullish: The Inference Companies

When the model is free, serving is the business

Inference providers are the direct monetization layer for open weights. They capture the serving revenue Moonshot gives up, and the private market has spent twelve months repricing them for it.

Fireworks closed a $1.5B Series D at $17.5B this week, up from $4B in late 2025, with ARR past $1B (5x YoY), daily token volume past 40T, and ~95% of tokens served on models fine-tuned from open weights. Together raised at $8.3B on more than $1.15B of bookings, serving open models exclusively. Baseten’s estimated ARR tripled to ~$600M in a single quarter. Nvidia agreed to pay ~$20B for Groq. Cerebras doubled on its IPO day.

Now watch the mechanism run in real time. K3 is currently served by exactly one provider: Moonshot itself. July 27 changes that. When K2.6’s weights opened, it hit #1 on OpenRouter with 1.88T tokens served in a week across roughly twenty competing hosts; when DeepSeek’s last flagship dropped, Fireworks listed it the same day. Hosts profitably served K2.7-class models (32B active parameters) at $4 per million output tokens. K3 activates 50B parameters and lists at $15. That spread is the auction’s opening bid, and competition will close it toward cost — margin that accrues to serving platforms and their GPU suppliers, not to Moonshot, which collects nothing and has already seeded vLLM with the attention kernels to accelerate its own disintermediation.

Who Wins & Why

The margin ledger, layer by layer

Call it margin gravity. In AI, margin flows down the stack until it pools at the scarcest physical constraint. Open frontier weights unbundle intelligence from the infrastructure that serves it — and what cannot be downloaded is gigawatts of contracted power, installed accelerators, serving economics won through utilization and kernels, and the enterprise control plane.

What to watch — five dated catalysts

1. July 27: the weights and the license. Confirmation of modified-MIT terms, and how fast third-party hosts light up. Precedent says same-day: Fireworks listed DeepSeek’s last flagship within hours of the drop.

2. A token-efficient open frontier model. Baker’s true Sputnik moment. K3 at $0.94/task vs. Grok 4.5 at $0.31 defines the gap; the first open model to close it ends the closed labs’ last economic argument below the frontier.

3. Grok 5, Composer 4, Muse 2 on the Pareto frontier. Vertical integration is the second road to model-layer margin compression. Multiple integrated players at multiple price-intelligence points would finish what open weights started.

4. The tokens-to-dollars gap. Open weights: 29% of gateway tokens, under 4% of spend. Each quarterly print that narrows it converts the token vote into revenue for inference providers and neoclouds — and out of lab pricing power.

5. How fast the labs become their own infrastructure. OpenAI’s 10GW Broadcom program and 6GW of AMD; Anthropic’s million TPUs and $30B Azure commitment. The faster the labs verticalize, the louder they are conceding that the model layer alone no longer carries the valuation.

The risk case is concentrated, not diffuse. Enterprise dollars stay closed longer than the token data implies — trust, liability, inertia. Washington moves: the House already probed Airbnb and Cursor’s parent over Chinese-model use, and restrictions in regulated sectors would slow adoption where the dollars are. Or a lab’s internal checkpoints compound into a lead nobody closes. Each slows the repricing. None reverses it, because the mechanism — margin flowing to the scarcest physical constraint — doesn’t depend on which model wins.

The weights are free. The gravity is not.

• One Blackwell GPU costs Nvidia roughly $6,400 to build — 45% of that is HBM memory alone — and sells for $30,000–$40,000. The 5–6x gap between those two numbers, captured entirely at the design layer, is the single clearest illustration of where value in this stack actually accrues.

• Six buyers (Microsoft, Amazon, Alphabet, Meta, Oracle, CoreWeave) will deploy over $760B of capex in 2026, up from $410B in 2025 — but margin capture is concentrated in five chokepoints: chip design (Nvidia, 75% gross margin), foundry (TSMC, 66%), lithography (ASML, 53%), memory (SK Hynix/Micron, 70–85%), and networking (Arista/Broadcom, 62–77%).

• Assembly and rental — Dell (11% op. margin), Supermicro (8% gross), CoreWeave (6% op. margin) — capture almost none of it despite growing revenue 106–346% year-over-year.

• HBM memory’s 70–85% margins are real but cyclical, not structural — a formal scoring framework below puts it at 10/15 on durability, versus 13–15/15 for Nvidia, TSMC, and ASML. DRAM contract prices are up ~820% year-over-year into 2026, the steepest spike in at least three prior four-year memory cycles, and NAND spot prices have already fallen below contract prices — a classic early crack that has historically preceded a cycle turn.

• CoreWeave’s capex-to-revenue ratio exceeds 250%, financed substantially with debt — a materially different risk profile than hyperscaler capex funded by 37–46% operating margins on an already-profitable core business.

• Inference cost per token has fallen ~99.7% since 2023, yet hyperscaler capex is up 77% in 2026 alone — the DeepSeek shock of January 2025 (Nvidia’s largest one-day market-cap loss in history) tested the “efficiency kills demand” thesis in real time and it failed; Jevons paradox, not substitution, has governed this cycle so far.

• Verdict: overweight the durable-moat tier (Nvidia, TSMC, ASML, Arista, hyperscale cloud); trade, don’t hold, HBM memory; avoid or underwrite carefully the commodity-assembly and neocloud tier.

Thesis

Roughly $760 billion of hyperscaler and neocloud capital expenditure will move through the AI infrastructure stack in 2026 — Microsoft, Amazon, Alphabet, Meta, Oracle, and CoreWeave alone. Add Stargate, sovereign AI programs, and enterprise on-prem buildouts, and the gross figure clears $1 trillion. Almost none of this is evenly distributed. It concentrates in a handful of chokepoints — one lithography company, one foundry, three memory makers, two networking suppliers, one cooling specialist — and it evaporates almost entirely by the time it reaches the companies bolting the boxes together and renting out the racks.

The closest historical analogue is the late-1990s fiber-optic buildout: enormous, demand-justified capital investment in physical infrastructure, a handful of equipment suppliers who captured durable value (Cisco), and a much larger population of network operators who overbuilt capacity, took on debt to do it, and were largely wiped out or absorbed when the financing cycle turned — even though the underlying thesis (the internet would need vastly more bandwidth) was completely correct. The lesson from that cycle was never “don’t invest in the buildout.” It was “know which layer of the buildout you own.” The cloud capex cycle of 2010–2020 offers the more benign counter-example: AWS, Azure, and Google Cloud absorbed a decade of heavy infrastructure capex and came out the other side as some of the highest-margin software franchises in history, because the demand side (enterprise workload migration) proved durable enough, and the supply side concentrated enough, that scale itself became the moat. This piece is, in effect, an attempt to sort every company in the current AI buildout into one bucket or the other before the cycle answers the question for us.

This is the central fact of the AI buildout that market-map slides obscure: value capture in this stack is not correlated with dollars flowing through a layer — it’s correlated with how replicable that layer is. Nvidia converts 75 cents of every revenue dollar into gross profit. Dell converts eleven. Both are “AI infrastructure companies.” They are not remotely the same business.

The physical constraint underneath all of this is increasingly power, not capital. U.S. data center power demand is projected to climb from 31 gigawatts in 2025 to 41 GW in 2026 and 66 GW in 2027 — more than doubling in two years — while global data center power demand rises 27% in 2026 alone, from 104 GW to 132 GW, en route to a projected 290 GW by 2030. U.S. data centers’ share of peak summer electricity demand is set to nearly double, from 4.1% in 2025 to 8.5% by 2027. Every company in this piece is, at some level, racing against a grid buildout that moves on a slower clock than capital markets do — which is precisely why Vertiv’s backlog and the broader power/cooling layer matter more than their market cap alone suggests.

The rest of this piece works through the stack layer by layer, with the 2026 numbers, to build a durable mental model for where profit pools sit today and which of them survive the next three to five years of capacity build.

The Stack by Market Value

Before getting into margin structure, it’s worth seeing the stack the way public markets currently price it. The dispersion is extreme: Nvidia alone is worth more than the bottom eleven companies in this piece combined.

Three things stand out. First, the design and foundry layers (Nvidia, TSMC) and the hyperscale cloud layer (Microsoft, Amazon) account for the overwhelming majority of aggregate value — over $12 trillion combined among just four companies. Second, the memory names (Micron, Samsung, SK Hynix) have re-rated dramatically over the past twelve months on the back of the HBM cycle, with Micron alone now larger than ASML despite a much shorter history of sustained profitability. Third, and most tellingly for anyone doing bottom-up diligence in this sector, the assembly and neocloud tier — Dell, Supermicro, CoreWeave, Crusoe — trades at a small fraction of the value of the layers directly above and below it in the physical supply chain, despite processing enormous and growing volumes of capital. The market has already priced the framework this piece is describing; the question is whether it has priced it correctly for what comes next.

How Big Does This Get

Sizing estimates for the buildout vary by scope but agree on direction and magnitude. Goldman Sachs’ baseline model implies $765 billion of annual AI capex in 2026, scaling to $1.6 trillion annually by 2031. McKinsey puts total data center capital requirements at $6.7 trillion through 2030, of which $5.2 trillion is AI-specific — roughly 70% of all data center capacity built over the rest of this decade will be AI-driven. Morgan Stanley’s more conservative frame estimates nearly $3 trillion of AI infrastructure investment flowing through the global economy by 2028, with more than 80% of that spending still ahead of it, and has separately floated a $40 trillion figure for the total economic value AI diffusion could eventually create — a number that dwarfs the infrastructure capex required to enable it by roughly an order of magnitude.

What matters isn’t which estimate is right. It’s that every credible forecast has AI infrastructure spending compounding through at least 2030, off a base that has already tripled in three years. That is the demand backdrop against which every margin and moat argument in this piece has to be read. The layers with pricing power aren’t betting on a spending peak arriving soon — they’re pricing a multi-year runway, and the sell-side keeps revising its own numbers up, not down.

The Efficiency Paradox

The best objection to this whole piece isn’t about any one company. It’s macro: if AI intelligence is getting radically cheaper, why does infrastructure spending keep accelerating? That question already got tested in the real world, so here’s what actually happened.

In January 2025, Chinese lab DeepSeek released a model claiming frontier-class performance trained for a reported $5.6 million — a figure that, whatever its precision, was low enough to detonate the market’s assumptions about compute intensity. Nvidia lost $589 billion of market capitalization in a single trading day, the largest one-day value destruction of any company in history, on the fear that “great angst as to the impact for compute demand” (Cantor’s phrasing at the time) meant peak GPU spending had arrived early. Eighteen months later, Nvidia’s market cap sits at $4.85 trillion — more than 8x its post-DeepSeek trough — and combined Big Four hyperscaler capex has gone from $410 billion (2025) to a guided $725 billion (2026), a 77% increase in the same window the market once feared would mark the top.

The token-cost side of that chart is dramatic on its own terms. a16z’s “LLMflation” analysis and industry API pricing both show GPT-4-class inference falling from roughly $30 per million tokens at launch (March 2023) to under $0.10 per million tokens for equivalent capability by 2026 — two orders of magnitude in three years, faster than PC compute or dot-com-era bandwidth cost declines historically ran. That’s one of the fastest unit-cost collapses of any general-purpose technology on record. Capex rose anyway, at an accelerating rate, in the same window.

The answer is Jevons paradox, playing out about as cleanly as it ever has: when the cost of something useful falls, people use so much more of it that total spending rises even as the unit economics collapse. Morgan Stanley Research puts a precise shape on how this plays out over the rest of the decade: the AI infrastructure TAM grows from roughly $315 billion in 2025 to nearly $1 trillion by 2030, and within that growth, inference compounds at a 42% CAGR between 2025 and 2030 versus a materially slower rate for training — enough to flip the mix from training-majority today (roughly 75% training / 25% inference in 2025) to inference-majority

by the end of the decade (56% inference / 44% training in 2030).

Training doesn’t shrink in absolute terms — it keeps growing, from $236 billion to a projected $429 billion — it simply stops being the terminal use case. Inference, priced per query at a small and falling fraction of its 2023 cost, becomes the larger and faster-compounding half of the market anyway, because there are vastly more queries than there are training runs, and grows nearly seven-fold ($79 billion to $546 billion) over the same window. That shift favors the durable-moat and hyperscale-cloud tiers over a pure training-compute thesis, because inference workloads are more distributed, more latency-sensitive, and — importantly for Broadcom’s and the custom-silicon cohort’s argument — more receptive to purpose-built, less flexible, cheaper-per-query silicon than training workloads ever were. The efficiency curve is real and it’s fast, and so far it has accelerated demand rather than killed it, for every layer in this stack except the ones already flagged as weak below.

Scarcity Sets the Margin, Not the Spend

Every layer in this stack is growing. Growth isn’t what separates the winners — margin structure is. The pattern repeats at every node of the chain.

Layers with structural scarcity — a single supplier, a multi-year lead time to add capacity, or an entrenched software/IP moat — hold gross margins of 60–85% almost regardless of end-demand volatility. Layers that are capital-absorbing but replicable — assembly, integration, undifferentiated compute rental — get bid down to single-digit or low-teens margins the moment more than one credible competitor shows up with a checkbook.

Nvidia (chip design), TSMC (advanced foundry), ASML (EUV lithography), and the HBM oligopoly (SK Hynix, Samsung, Micron) all sit in the first category. Dell, Supermicro, and the neoclouds (CoreWeave, Crusoe) sit in the second. Networking (Arista, Broadcom) and power/cooling (Vertiv) sit in a durable middle tier: real engineering moats and real pricing power, but smaller absolute scale than the chip or foundry layers.

That spread is the real story. A 12-point gap between the top of the stack (77% gross margin at Broadcom’s custom silicon business) and the bottom (6% operating margin at CoreWeave, 8% gross margin at Supermicro) isn’t noise. It’s the market pricing scarcity correctly, layer by layer, in real time.

To make that judgment reproducible rather than impressionistic, score every layer on the three factors that actually determine whether a margin survives a capacity cycle: supply scarcity (how many credible suppliers exist today), switching cost / IP moat (what it costs a customer to leave), and capital intensity as a barrier (whether the capex required to enter is itself a deterrent, or merely a checkbook problem for a well-funded entrant). Score each 1 (weak) to 5 (strong) and the ranking falls out of the arithmetic rather than the narrative.

Two results cut against the size-based intuition of a market-cap ranking. First, HBM memory scores a 10/15 — respectable, but nearly three points below Nvidia and five below ASML — because its switching-cost score is only a 2: HBM is a qualified, multi-sourced commodity component, and a hyperscaler buying HBM3E today is not locked into SK Hynix the way it is locked into CUDA. Second, hyperscale cloud outscores Nvidia’s design layer on switching cost and capital intensity (5 and 5, versus Nvidia’s 5 and 3), a reminder that Nvidia’s moat is almost entirely a scarcity-and-software story, while AWS and Azure’s moat is scarcity-light but capital- and switching-cost-heavy. Both produce durable margin, but through different mechanisms — which matters for how fast each moat could erode, and under what conditions.

Layer 1 — Design

Nvidia earns roughly 75% gross margin. Dell earns 11% operating margin on its AI servers. Every allocator eventually asks why, and the answer is structural, not incidental.

Nvidia reported Q1 FY2027 (quarter ended April 2026) revenue of $81.6 billion, up 85% year-over-year, with data center revenue of $75.2 billion, up 92%. GAAP gross margin was 74.9%. Operating income was $53.5 billion — a 65.6% operating margin — and net income was $58.3 billion, up 211% year-over-year, on free cash flow of $48.6 billion. The company’s market capitalization sits at roughly $4.85 trillion, making it the most valuable company on earth.

Nvidia doesn’t manufacture a single wafer. It designs silicon and owns CUDA, the software layer that has absorbed fifteen years of developer tooling, libraries, and institutional muscle memory. That software moat is what lets Nvidia charge a price for the H100/B200/Rubin generations that reflects the value of the entire AI training and inference workflow, not the marginal cost of a chip. Demand exceeds supply at every node TSMC can produce; Nvidia is the scarce resource, and it prices like one.

This scarcity cuts both ways. Nvidia’s own disclosures show four direct customers each representing more than 10% of total revenue in Q3 FY2026 — 22%, 15%, 13%, and 11% respectively, a combined 61% of revenue from four unnamed hyperscaler-scale buyers. That concentration is the natural consequence of selling the scarcest input in the stack to the handful of companies large enough to buy it at scale. But it also means Nvidia’s growth trajectory is now more sensitive to the capital-allocation decisions of four boardrooms than diversified revenue bases usually are — the same four boardrooms, in some combination of Microsoft, Amazon, Meta, and Google, that are simultaneously funding the custom-silicon programs built to reduce that dependency over time.

Dell’s Infrastructure Solutions Group (ISG), by contrast, posted a record $29 billion in Q1 FY2027 revenue with AI-optimized server revenue of $16.1 billion — up 291.7% year-over-year — against an ISG operating margin of just 10.5%, itself up only 80 basis points despite AI server volume growing nearly 800%. Dell closed FY2026 with $64.1 billion in cumulative AI orders and a $43–51 billion AI backlog. The volume is real and enormous. The margin is not moving.

The mechanism is simple: Dell (and Supermicro, whose gross margin has run between 6.3% and 10.1% across recent quarters despite $38.9–40.4 billion in FY2026 revenue guidance) buy the same Nvidia GPU, the same SK Hynix HBM, the same Broadcom NIC that every other integrator can buy, and assemble them into a box. There is no proprietary IP in the assembly step that a customer can’t get from three other vendors within a quarter. Dell and Supermicro are price-takers on their most expensive input (the GPU) and price-takers on their output (the finished server) — squeezed from both sides in a business where the customer, a hyperscaler or enterprise, can and does multi-source.

It’s the cleanest illustration in this whole framework: identical end demand, identical growth rate, radically different margin, because one layer is a monopoly-adjacent software platform and the other is a commodity assembly line.

The clearest way to see this is to follow a single chip from the fab to the rack. Epoch AI’s cost-breakdown analysis puts total manufacturing cost (COGS) for one Blackwell B200 GPU at roughly $6,400 — and within that figure, HBM memory alone is 45% ($2,900), ahead of the logic die itself (22%, $1,400), CoWoS advanced packaging (17%, $1,100), and substrate/testing (16%, $1,000). Nvidia then sells that same GPU for a street price in the $30,000–$40,000 range: a five-to-six-times markup captured entirely at the design layer, on top of a component cost structure in which the memory supplier has already embedded its own 70%+ gross margin before the chip is even assembled.

Scale that to a full GB200 NVL72 rack — 72 of these GPUs plus Grace CPUs, NVLink switching, and cooling infrastructure — and the system-level price runs to roughly $3.1 million for the compute portion, $3.9 million all-in once networking and storage are included. Dell or Supermicro’s job is to take that $3.9 million of mostly Nvidia-priced, SK-Hynix-priced, and Broadcom-priced components, add chassis, cabling, integration labor, and a support contract, and mark the whole thing up by single digits. Every dollar of margin in that rack was captured before the box was assembled. Assembly is where margin goes to be spent, not made.

The Layer Beneath the Layer

Nvidia’s 75% gross margin is a hardware number attached to a software moat, and the two are worth separating, because software is the layer in this stack most resistant to every competitive pressure described elsewhere here. CUDA isn’t one product. It’s roughly two decades of accumulated libraries, compilers, and developer tooling (cuDNN, NCCL, TensorRT, and hundreds of downstream frameworks) that make Nvidia silicon the default target for every new model architecture before anything else gets considered. That accumulated software surface is precisely what custom silicon programs at Google, Amazon, and Microsoft have to replicate from a standing start, and why, even with $225 billion of committed Trainium spend, Amazon’s own leadership frames the custom-silicon push as a workload-specific complement to Nvidia dependency rather than outright replacement on any near-term timeline. Beneath CUDA sits a second software layer that gets even less investor attention: the orchestration, scheduling, and cluster-management tooling (Slurm, Kubernetes-based GPU schedulers, Ray) that determines whether a given cluster actually achieves the utilization its hardware spec implies. A cluster running at 55% effective utilization instead of 85% is, economically, a cluster with roughly a third of its capital sitting idle, which makes this orchestration layer, though it captures little margin directly today, one of the more interesting adjacent software opportunities the current buildout is creating.

The strongest bear case on the CUDA moat is worth taking seriously, because it’s the biggest swing factor in whether Nvidia’s design-layer score stays a 13/15 or drifts toward Broadcom’s contested 9/15 over a five-year horizon. PyTorch, the framework most model training now runs through, has spent several years deliberately abstracting away direct CUDA calls behind its own compiler stack (torch.compile, TorchInductor); OpenAI’s Triton language does the same for custom kernel authorship, letting a developer write hardware-agnostic code that can, in principle, target AMD’s ROCm or a custom ASIC’s compiler backend with far less rewrite effort than five years ago. If that abstraction layer keeps maturing, the actual switching cost of leaving Nvidia silicon falls even as CUDA itself stays dominant, because the thing developers depend on stops being CUDA specifically and starts being the compiler layer sitting on top of it. The counter-evidence: this transition has been “underway” for essentially the entire period Nvidia’s data center revenue compounded from single-digit billions to $75 billion a quarter, and every attempted software abstraction to date has still shipped its best-optimized path on Nvidia hardware first. The moat looks real today and slowly eroding at the edges — not collapsing, not permanent — which is why the framework scores it a 5 on switching cost while flagging it as the dimension most likely to move.

Layer 2 — Foundry

TSMC sits at the foundry chokepoint of the stack, and the bottleneck inside it has a name: CoWoS.

TSMC posted Q1 2026 revenue of $35.9 billion, up 40.6% year-over-year, at a record 66.2% gross margin, 58.1% operating margin, and 50.5% net margin. Full-year 2026 revenue is guided to grow more than 30% in USD terms. Market capitalization: roughly $2.1 trillion.

TSMC’s moat isn’t really “advanced lithography” — that’s ASML’s layer, one step upstream. TSMC’s moat is the accumulated manufacturing yield, process know-how, and packaging capacity that no other foundry has matched at volume. The specific chokepoint that determines how much AI silicon can physically reach the market in any given quarter is CoWoS (chip-on-wafer-on-substrate), the advanced packaging process that stitches GPU compute dies to HBM stacks.

CoWoS capacity has gone from roughly 35,000 wafers per month at the end of 2024 to about 75,000 at the end of 2025, with a target of 125,000–130,000 by the end of 2026 — nearly a 4x expansion in two years. TSMC’s own CEO described the line as “sold out through 2026” as of the company’s June 2026 annual meeting. It’s the most important supply constraint in the entire AI stack: it doesn’t matter how many GPUs Nvidia can design or how much HBM SK Hynix can fab if CoWoS capacity can’t marry the two together. TSMC captures margin here not because it competes on price, but because for the 3-nanometer/2-nanometer AI accelerator category, there is effectively no alternative supplier at scale. Samsung Foundry and Intel Foundry remain multiple process generations and multiple years of qualification behind on both logic and advanced packaging.

The Layer Investors Overlook: Optics

Every GPU cluster large enough to train a frontier model is, physically, an interconnect problem before it’s a compute problem. Copper reaches its practical bandwidth-distance limit within a single rack; everything beyond that — GPU-to-GPU across racks, rack-to-rack across a data hall — runs over optical transceivers. As cluster sizes have scaled from thousands to hundreds of thousands of GPUs, the optics bill of materials per cluster has scaled with it: a single large training cluster today consumes hundreds of thousands of 800G and increasingly 1.6T optical transceivers, a cost line that barely existed at this scale three years ago. This is a growing profit pool, occupied by transceiver and silicon-photonics suppliers sitting adjacent to Broadcom’s and Arista’s networking businesses, and it’s also a genuine physical bottleneck: optics manufacturing capacity, like CoWoS, cannot be added instantly, and a shortage here caps effective cluster size regardless of how many GPUs and how much HBM are sitting in inventory. It’s the least-discussed chokepoint in this entire stack relative to how much it actually matters.

Layer 3 — Lithography

One step further upstream sits ASML, the Dutch company that is the sole global supplier of EUV (extreme ultraviolet) lithography systems — the machines TSMC, Samsung, and Intel all need to print anything below the 7nm node. ASML’s Q1 2026 net sales were €8.8 billion at a 53% gross margin; full-year 2026 guidance calls for €34–39 billion in revenue (roughly 11.6% growth at the midpoint) with 51–53% gross margin. Backlog stood at €38.8 billion at year-end 2025 after a record €13.2 billion bookings quarter. Market capitalization: approximately €600B / $690 billion.

ASML’s margin is the lowest of the “scarcity” tier in this piece, not because its moat is weaker, but because its business model runs on multi-year order books and long manufacturing lead times rather than the vertiginous spot-market dynamics of GPUs or memory. It is, structurally, the most durable monopoly in the entire chain: there is no second EUV supplier, full stop, and there won’t be one for the foreseeable future given the two-decade, multi-hundred-billion-dollar R&D investment embedded in the current machines. ASML’s growth rate looks modest next to Nvidia’s or Micron’s because it doesn’t need to chase the AI cycle. Every advanced chip on earth, AI or otherwise, already runs through its tools.

Layer 4 — Memory

High-bandwidth memory (HBM), the memory stacked directly onto the GPU package to feed it data at the bandwidth AI training requires, has been the best-performing sub-sector in semiconductors in 2026, and it deserves the most caution of anything in this piece.

SK Hynix dominates the category, with roughly 50–56% global HBM share, reported Q1 2026 revenue of approximately ₩52.6 trillion (~$35.5 billion), up 198% year-over-year, at an operating margin near 72%. Its market cap has roughly doubled over the trailing twelve months to around $586 billion. Micron has ridden the same wave from the number-three HBM position (roughly 20–24% share): fiscal Q3 2026 revenue hit $41.46 billion, up 346% year-over-year, at an 84.9% non-GAAP gross margin, with FQ4 guidance of $50 billion in revenue at roughly 86% gross margin — a business that traded at single-digit multiples on commodity-cycle earnings as recently as 2023 now carries a $1.3 trillion market cap. Samsung holds the remaining roughly 28% and is racing to qualify HBM4 for the next Nvidia platform generation.

Samsung is the wild card inside this layer, because it’s the only company in the entire stack playing on two boards at once: it holds roughly 28% of HBM share directly behind SK Hynix and Micron, and it’s simultaneously the only credible, if distant, alternative to TSMC in advanced-node logic foundry. That dual exposure makes Samsung the best-positioned incumbent to benefit if either the memory oligopoly holds or TSMC’s packaging bottleneck creates enough overflow demand for a qualified second source, and the most likely company to absorb margin if HBM pricing normalizes before Samsung’s foundry business has scaled enough to offset it. At a $1.34 trillion market cap, Samsung trades as a memory story today; the foundry option value is, for now, priced closer to zero.

The margins here, 70–85% gross, look identical to Nvidia’s on paper. They aren’t the same kind of margin. Nvidia’s margin is protected by a software ecosystem that took fifteen years to build and that customers cannot easily leave. HBM’s margin is protected by a supply shortage: three suppliers, all racing to add capacity, in a category that has been through boom-bust cycles roughly every four years since DRAM existed as a product. Peak-to-trough swings recur on an almost metronomic four-year cadence, with prior cyclical turns clustering around 2010–2014, 2014–2018, and 2018–2022. SK Hynix, Samsung, and Micron are all expanding HBM capacity aggressively in response to today’s pricing, and capacity additions of this kind, in memory, have never once failed to eventually overshoot demand.

What makes the current cycle different is its slope, not its shape. Morgan Stanley Research and TrendForce data show DRAM contract pricing up roughly 820% year-over-year heading into 2026 — a spike that dwarfs any prior peak in at least three previous cycles, and the kind of move that historically has marked a cycle’s most extended phase rather than its middle innings.

Two additional signals from the same data cut in different directions. First, supply-chain inventory — measured in weeks of stock held across DRAM/NAND suppliers, PC OEMs, module makers, server makers, and smartphone makers — peaked in mid-2024 to early 2025 and has been normalizing through 2026, which removes the destocking buffer that cushioned prior cycles and means fresh demand or supply news now hits pricing more directly. Second, and more tellingly, spot and contract prices have decoupled in opposite directions across the two memory types. DDR5 spot pricing (~$45) currently runs ahead of contract pricing (~$37.50), consistent with a market still tightening in real time, while 512GB NAND/SSD spot pricing (~$84) has fallen below contract pricing (~$118) — an inversion that typically shows up when channel buyers start pulling back faster than the long-lead-time OEM contracts they’re locked into can adjust. That NAND inversion isn’t proof the cycle has turned. It’s the kind of small, early crack in the data that historically precedes one, and it’s the clearest reason the 2028 mean-reversion call earlier in this piece is a base case rather than a low-confidence guess. The 2026 HBM margin structure is real cash flow today and a legitimate multi-year window, but it’s a cyclical profit pool wearing the appearance of a structural one, and it’s the layer in this piece most likely to see today’s 70%+ margins compressed first.

Layer 5 — Networking and Custom Silicon

Broadcom occupies an unusual dual position. Its legacy networking/semiconductor business carries company-wide gross margins around 77% (down slightly from 79.1% a year earlier as mix shifts). Its faster-growing custom AI silicon (XPU) business, designing application-specific accelerators for hyperscalers who want Nvidia-alternative compute, generated $8.4 billion in AI revenue in Q1 FY2026, up 106% year-over-year, against a $73 billion AI backlog and a company target of $100 billion in annual AI semiconductor revenue by 2027. Market capitalization has moved between roughly $1.8 and $2.3 trillion through 2026 depending on the week, among the most volatile large-cap valuations in the stack, reflecting genuine investor uncertainty about how much of Nvidia’s design-layer margin Broadcom’s custom-ASIC model can ultimately capture.

Arista Networks is the purer comparison for the “networking has real pricing power” argument: FY2025 revenue of $9.0 billion (up 28.6%), FY2026 guidance raised to roughly $11.5 billion (27.7% growth), and gross margin held steady at 62–64% even as AI-cluster interconnect demand strains the entire supply chain. Arista’s moat is software-defined switching (EOS) plus deep incumbency in the largest cloud and AI cluster deployments, a smaller-scale, higher-durability cousin of Nvidia’s CUDA lock-in, at a $220 billion market cap.

Custom silicon is the most important competitive threat to Nvidia’s margin structure in the entire stack, and Broadcom is the company best positioned to monetize that threat, which is exactly why its market cap has been so volatile. It’s simultaneously a bet for the Nvidia alternative and against Nvidia’s durability.

That threat is no longer speculative. Custom AI ASIC shipments are projected to grow 44.6% in 2026, nearly three times the 16.1% growth rate projected for merchant GPUs, as Google’s Ironwood TPU, Amazon’s Trainium 3, Microsoft’s Maia 200, and Meta’s MTIA all ship in volume. Amazon has disclosed more than $225 billion in Trainium revenue commitments, and its custom-silicon program reportedly crossed a $20 billion annualized run rate in Q1 2026 (largely internal transfer pricing to AWS rather than external merchant sales, but real capital allocation nonetheless). Broadcom and Marvell together control an estimated 95% of the custom-ASIC co-design market, which means the “threat to Nvidia” and “the next leg of Broadcom’s growth” are, functionally, the same trade.

None of this displaces Nvidia in the near term. CUDA’s ecosystem lock-in means workload migration to custom silicon happens model generation by model generation, not quarter by quarter. But it caps the multiple the market should be willing to assign to the durability of Nvidia’s 75% margin, even as the level of that margin stays intact for the foreseeable future.

Layer 6 — Power and Cooling

Vertiv is the cleanest “picks and shovels” business in the stack: data center power and thermal management, benefiting from the industry’s forced pivot to liquid cooling as rack power densities go vertical. The industry-average rack draws roughly 7.6kW. Nvidia’s GB200 NVL72 draws 120–140kW — 16 to 18 times the legacy average — and the 2026 Vera Rubin generation (VR200 NVL72) is specced at 190–230kW, with the long-context CPX variant near 370kW. The 2027 Rubin Ultra (NVL576) roadmap points to roughly 600kW per rack, which is why Nvidia is simultaneously pushing the industry toward 800-volt DC facility power architecture just to make racks north of 600kW physically feasible. Air cooling can’t dissipate any of this. Liquid cooling isn’t an option at these densities; it’s a requirement, and that requirement is what turns Vertiv’s backlog into multi-year, engineering-locked revenue rather than a commodity hardware order. Q1 2026 revenue was $2.65 billion (up 30.1% year-over-year); full-year guidance was raised to $13.5–14.0 billion; adjusted operating margin expanded 430 basis points to 20.8%; backlog more than doubled to over $15 billion, covering 12–18 months of forward revenue. Market cap: roughly $121 billion.

Vertiv’s margin sits well below the chip layer but meaningfully above the assembly layer, for a specific reason: power and cooling infrastructure is engineered and specified, not commoditized. It requires deep integration with a data center’s electrical and mechanical design, creates real switching costs once installed, and has a backlog structure that gives multi-quarter revenue visibility other layers in the stack lack. It’s a smaller moat than the chip layer, but a durable one — a fact the market underappreciated through 2023–2024 and has since substantially repriced.

Layer 7 — Cloud and Neocloud

Hyperscale cloud and neocloud look like the same business — renting AI compute — and this is where the framework produces its sharpest, most investable distinction: they are not, financially, the same business at all.

AWS posted Q1 2026 segment revenue of $37.6 billion (up 28% year-over-year) and segment operating income of $14.2 billion, a 37.7% operating margin. Microsoft Azure grew 39% year-over-year in constant currency in the same period, with company-wide operating margin at 46.3%, up from 45.7% a year prior, even as Microsoft guided to roughly $190 billion of calendar-2026 capex (up 61% from 2025). Oracle Cloud Infrastructure (OCI) grew 93% to $5.8 billion in its most recent quarter, backed by a staggering $638 billion remaining-performance-obligations backlog (up 363% year-over-year), but Oracle’s implied margin on this business is thin, and the stock gave back roughly a third of its market cap between early June and early July 2026 (from ~$614 billion to ~$450 billion) as investors reassessed how much of that backlog converts to profitable revenue versus GPU-lease pass-through.

CoreWeave, the largest pure-play “neocloud,” tells the cautionary version of this story in numbers: 2026 revenue guidance of $12–13 billion against a $99.4 billion contracted revenue backlog, 2026 capex guidance of $31–35 billion, roughly $25 billion of debt on the balance sheet as of Q1 2026, and an adjusted operating margin of only about 6%. Crusoe, still private, is scaling revenue toward an estimated $2 billion in 2026 and has seen its valuation nearly triple in eight months, from just over $10 billion (October 2025 Series E) to reported talks near $30 billion (July 2026) — a re-rating pace that outstrips even the memory names.

The mechanism behind the margin gap is depreciation and financing structure, not customer demand. Hyperscalers depreciate GPU capex over five-plus years against a diversified, high-margin software/services revenue base and can subsidize AI infrastructure economics with the rest of the business. Neoclouds are frequently financing GPU purchases with debt collateralized by the GPUs themselves, running at far higher balance-sheet leverage, and monetizing a single asset class (raw compute rental) into a market where hyperscalers are simultaneously their landlord, their supplier, and increasingly their competitor.

CoreWeave’s $99 billion backlog is the bull case for the neocloud model: real, signed, multi-year demand from credible counterparties, largely Microsoft, OpenAI, and other frontier labs. Its $25 billion of debt against a 6% operating margin is the bear case: neoclouds have taken on hyperscaler-level capital intensity without hyperscaler-level margin protection, and CoreWeave’s market cap has fallen from prior highs to roughly $47–58 billion through June 2026 as the market repriced that mismatch.

Growth Is Universal; Margin Is Not

Isolate growth alone and every layer in this stack looks like a buy. That’s exactly the trap.

Dell’s AI server business (+292%) and Micron’s memory business (+346%) are growing faster than Nvidia (+85%) or TSMC (+41%). By a naive growth-rate screen, the assembly and memory layers look like better businesses than the design and foundry layers. They aren’t, because growth in this stack is a function of end-market demand, which every layer shares roughly equally, while margin is a function of competitive structure, which varies enormously by layer. A commodity business growing 292% is still a commodity business — it simply commoditizes a larger dollar volume every quarter. It’s the most common mistake in surface-level analysis of the AI infrastructure trade, and it’s why the margin-by-layer view earlier in this piece matters more than any growth chart on its own.

Capital Intensity

The other variable growth-rate screens miss is how each layer’s expansion is financed, and this is where the durable and temporary tiers diverge most sharply in risk, not just in margin.

TSMC and ASML both run enormous capital expenditure programs — TSMC’s annual capex runs in the $40–52 billion range — but they fund it from operating cash flow generated at 50%+ net margins, leaving the balance sheet essentially unlevered relative to earnings power. Hyperscalers are similar in kind if not in scale: Microsoft’s ~$190 billion 2026 capex plan and Amazon’s roughly $200 billion figure are both funded by companies with 37–46% operating margins on existing, diversified, high-margin revenue bases. AI capex is large in absolute terms but small relative to the cash-generating power of the businesses funding it.

CoreWeave is the inverse case, and the clearest illustration of why capital intensity is a risk factor independent of growth or even margin. Against 2026 revenue guidance of $12–13 billion, the company is guiding to $31–35 billion of capex, a ratio north of 250%, financed substantially with debt (roughly $25 billion outstanding as of Q1 2026) collateralized by the GPUs the capex buys. That’s a fundamentally different risk profile than a hyperscaler spending a similar dollar amount out of a profitable core business: CoreWeave’s downside case involves creditors, not just a lower stock multiple, if its $99.4 billion revenue backlog fails to convert to cash flow on schedule. Oracle’s OCI buildout sits somewhere in between: funded by a large, profitable database and applications franchise, but increasingly reliant on customer prepayments to bridge the gap between capex outlay and recognized revenue, a structure that inflated Oracle’s own reported capex by $20–25 billion in FY2026 and that the market punished with a roughly 27% pullback in market cap between early June and early July 2026 alone.

The rule of thumb for allocators: the safest capital intensity in this stack is capital intensity funded by an already-profitable, already-diversified business. The riskiest is capital intensity funded by debt against a single, undifferentiated, cyclically exposed asset class. That’s the CoreWeave and Crusoe model, and it’s why both companies’ market caps and funding structures deserve more scrutiny than their revenue growth rates alone would suggest.

Putting It Together: Margin vs. Growth Across the Whole Stack

Pricing power (upper right) — Nvidia, Micron, SK Hynix, Broadcom’s AI business. High growth and high margin. The risk here is duration: for the memory names in particular, today’s margin is a function of a supply shortage, not a permanent structural advantage.

Scarce and slow (upper left) — ASML, TSMC, Arista, hyperscale cloud. High margin, comparatively modest growth. These are the most durable profit pools in the stack precisely because they aren’t chasing the AI cycle; AI demand is additive to an already-entrenched, already-profitable core business.

Commodity growth (lower right) — Dell ISG. Enormous volume growth, thin and barely-improving margin. Clear evidence that revenue growth and value capture have become decoupled at the assembly layer.

Structurally squeezed (lower left) — Supermicro, CoreWeave, Oracle OCI (on an implied-margin basis). Real revenue, real demand, and a cost structure (component costs, GPU depreciation, debt service) that consumes nearly all of it.

Where the Money Is Actually Coming From

None of this margin analysis matters without sizing the capital flow behind it. Big Tech capex for AI infrastructure now runs at a scale that exceeds the GDP of most countries.

Microsoft, Amazon, Alphabet, and Meta combined are on pace for roughly $725 billion of 2026 capital expenditure, up from about $410 billion in 2025 — a 77% increase in a single year. Layer in Oracle’s roughly $50 billion (a figure that understates cash capex by $20–25 billion once customer GPU prepayments are excluded) and CoreWeave’s $31–35 billion, and six buyers alone deploy over $760 billion into this stack in 2026. Consensus sell-side estimates for total hyperscaler AI capex in 2026 sit at or above $500–527 billion even before neoclouds and sovereign programs (Stargate’s initial $100 billion tranche of a targeted $500 billion by 2029) get added in.

That’s the scale of demand pull that explains why every layer above the assembly tier is capacity-constrained rather than demand-constrained. The binding question for every company in this piece isn’t “will demand arrive” — it already has, several times over, in the form of signed backlog. The binding question is which layers can convert that demand into margin that survives the next capacity cycle.

Where Competition Is Intensifying — and Where It Isn’t

Four fronts to watch, ranked by how much they threaten today’s margin structure:

Custom silicon vs. Nvidia (highest threat to the top of the stack). Google’s TPU program, Amazon’s Trainium, Meta’s MTIA, and the broader Broadcom/Marvell custom-ASIC design business represent the only credible long-term pressure on Nvidia’s 75% gross margin. Broadcom’s $73 billion AI backlog and $100 billion 2027 target are the clearest market signal that hyperscalers are willing to spend real design dollars to reduce Nvidia dependency — not because CUDA can be replicated quickly, but because at hyperscaler volume, even a partial workload shift to purpose-built silicon changes the unit economics enough to justify the multi-year design investment.

Memory capacity response (highest threat to the current HBM supercycle). SK Hynix, Samsung, and Micron are all expanding HBM capacity in direct response to 2025–2026 pricing, and all three have publicly committed to next-generation HBM4 qualification timelines that pull forward the point at which combined supply catches demand. Every previous memory upcycle in the history of the DRAM industry has ended the same way, with the suppliers’ own capacity response overshooting demand and collapsing pricing, typically within 18–30 months of the cycle’s peak signing activity. Nothing suggests this cycle breaks that pattern structurally; there is only evidence that AI training and inference token volumes are growing fast enough to have kept HBM undersupplied for longer than earlier cycles, and that the qualification complexity of HBM4 (versus commodity DDR) may modestly extend the time it takes new capacity to actually count as usable supply. Directionally, though, three well-capitalized competitors racing to add capacity into a 70%+ gross margin is about as reliable a mean-reversion setup as semiconductors produce.

Neocloud commoditization (highest threat to the bottom of the cloud layer). CoreWeave, Crusoe, Lambda, and a growing list of GPU-rental businesses are underwritten by the same asset (Nvidia GPUs) financed against the same buyer pool (frontier AI labs and hyperscaler overflow demand). As hyperscalers build out their own capacity and become less reliant on neocloud overflow, and as more entrants chase the same financing structure, this layer faces the fastest margin compression risk of anything in the stack outside of memory.

Foundry and packaging alternatives (lowest near-term threat, highest long-term uncertainty). Samsung Foundry and Intel Foundry are both investing to close the gap with TSMC on both leading-edge logic and advanced packaging, and both are multiple years and multiple process nodes behind at AI-relevant volume. Geopolitical risk (Taiwan) is the more relevant near-term variable here than competitive risk. CoWoS capacity being “sold out through 2026” is a supply problem TSMC’s competitors would need years to meaningfully address even if they matched the technology today. A Taiwan-related supply disruption wouldn’t shift share to a competitor so much as it would remove 90%+ of the world’s leading-edge AI chip supply from the market simultaneously — a tail risk with no meaningful hedge anywhere else in this stack, and an argument that TSMC’s margin premium is still too low relative to the systemic risk it’s underwriting for the entire industry.

Moat Durability

This chart matters more than any other in this piece for allocators, because market capitalization is a poor proxy for moat durability across this stack. Broadcom is larger than Arista by market cap, but Arista’s switching-software moat is arguably more durable than Broadcom’s customer-co-design moat, which depends on hyperscalers continuing to prefer bespoke silicon partnerships over vertical integration. Micron is a larger company than Vertiv, but Vertiv’s engineered, backlog-protected power/cooling business is structurally more durable than Micron’s HBM margin, which is a function of a supply shortage rather than an unassailable technology lead.

The durable tier — ASML, TSMC, Nvidia, hyperscale cloud, Arista, Vertiv — shares a common feature: replicating any of them requires either a monopoly-grade technology position, a decade-plus of accumulated manufacturing yield, an entrenched software ecosystem, or contractually embedded switching costs. The temporary tier — HBM suppliers, Broadcom’s custom-ASIC business, and especially Dell/Supermicro and the neoclouds — shares the opposite feature: their current margin is a function of a supply/demand imbalance (memory, GPUs) or a temporary strategic preference (custom co-design), not a barrier competitors cannot eventually cross.

“Durable” doesn’t mean these companies’ growth rates stay elevated forever. ASML’s ~12% growth this year already shows that durable-moat businesses can grow slower than the cycle’s hottest names while still compounding capital more reliably. It means the rate of return on the next incremental dollar of capacity stays high, because a competitor can’t show up in eighteen months and bid it away. That distinction, durability of returns versus rate of growth, is the one growth-rate-only screens systematically miss. It’s also why this framework treats Broadcom’s custom-silicon business (real growth, contested durability) so differently from Arista’s switching business (slower growth, largely uncontested durability), despite both sitting in the same “networking-adjacent” bucket on a superficial market map.

Implications

For venture and growth investors: the layers worth funding are the ones where a new entrant can either attack an incumbent’s replicability weakness (better financing structures or differentiated workload specialization for neoclouds, alternative packaging/interconnect approaches that ease the CoWoS bottleneck, novel cooling architectures that outpace Vertiv’s engineering lead), or become the next layer’s version of CUDA: a software or orchestration moat that makes switching costly regardless of whether the underlying hardware is commoditized. Capital deployed into “another GPU cloud” or “another server integrator” without either of those two attributes goes into the least defensible part of the entire value chain, regardless of how fast the segment’s revenue is growing.

For public-market allocators: the framework argues for structural overweight to the durable-moat tier (ASML, TSMC, Nvidia, Arista, hyperscale cloud parents) funded by structural underweight to the commodity-assembly and neocloud tier (server OEMs, pure-play GPU rental), with HBM memory treated as a high-conviction trade, not a long-term holding. The margin is real and probably persists through 2027, but memory cycles do not skip a beat forever, and 70–85% gross margins on DRAM-derived products have never once proven permanent.

For corporate strategists inside hyperscalers and enterprises: the custom-silicon investment thesis is validated by Broadcom’s backlog growth, but the CUDA software moat means the payback period on internal silicon programs is measured in years of workload migration, not quarters, and the capex commitment (Big Four at $725 billion in 2026) suggests the industry is underwriting Nvidia’s near-term pricing power even while actively trying to escape it. Enterprises buying AI infrastructure directly rather than through a hyperscaler should weight the capital-intensity framework above as heavily as the headline price-per-GPU-hour: a neocloud offering below-market pricing today is very often financing that discount with leverage that carries more risk than it appears to, and a vendor’s balance sheet is now a legitimate infrastructure-procurement diligence item in a way it simply wasn’t three years ago.

On timeline: the durable-tier thesis (Nvidia, TSMC, ASML) is a multi-year holding period call with limited need for precise entry timing, given the underlying scarcity is structural rather than cyclical. The HBM memory trade has a shorter effective horizon, probably 12 to 24 months of further upside before capacity additions from all three suppliers begin to bite, based on the historical cadence of prior memory cycles. The neocloud and assembly-layer caution flagged throughout this piece is closer to an immediate positioning call than a multi-year thesis: the capital-structure risk in that tier is already visible in 2026 pricing (CoreWeave and Supermicro’s compressed market caps relative to revenue growth are the market’s own early verdict), not a risk that requires a future catalyst to materialize.

Summary

Value in the AI infrastructure stack accrues to scarcity, not spend. Nvidia, TSMC, and ASML sit at genuine chokepoints — design IP with a software moat, manufacturing yield with no substitute at volume, and a lithography monopoly with no second supplier — and they price accordingly, at 53–75% gross margins that have held up across multiple demand shocks. HBM memory suppliers currently sit alongside them on margin (70%+ gross) but not on durability: their profit pool is a supply-shortage artifact that history says will eventually mean-revert. Networking (Arista) and power/cooling (Vertiv) occupy a smaller but genuinely durable middle tier, protected by software lock-in and engineering/backlog moats respectively. Server assembly (Dell, Supermicro) and neocloud GPU rental (CoreWeave, Crusoe) sit at the bottom of the stack by design: undifferentiated, multi-sourced, and priced down to single-digit or low-teens margins regardless of how fast their revenue grows, with the neocloud cohort carrying the extra risk of hyperscaler-grade leverage without hyperscaler-grade margin protection.

The $760 billion-plus flowing through this system in 2026 will keep growing. Who keeps it won’t change nearly as much as the market map suggests, and the parts of the map most likely to change are exactly the parts, like HBM memory and neocloud rental, that look most attractive on a trailing-twelve-month growth chart today.

For a decade the AI narrative was a compute narrative. The scarce resource was FLOPS, the winner was whoever owned the accelerator, and the network was plumbing. That framing is now obsolete. In a modern AI factory, a single training job spans tens of thousands of GPUs that must behave as one machine, and the fraction of wall-clock time those GPUs spend waiting on each other — not computing — is set almost entirely by the network. The interconnect has become the second silicon.

This piece maps the whole stack — the physical and logical layers that turn a warehouse of accelerators into a coherent supercomputer — and reads it the way an investor should. The consensus that networking is “the new moat” is directionally right but mispriced. The durable value is concentrating in three places a GPU-centric analysis tends to miss: the software that converts rated bandwidth into delivered bandwidth, the upstream optical supply chain everyone is capacity-constrained on, and the reliability layer that decides whether a 100,000-GPU run finishes at all. Compute is a spec sheet; the network is an operational outcome. Start with the map.

Figure 1 · The three domains of AI networking. Scale-up binds a handful-to-a-thousand accelerators into one memory space at terabytes/second over copper; scale-out lashes thousands-to-millions of GPUs into a cluster over optical Ethernet or InfiniBand; scale-across stitches AI factories together across buildings and metros. Each domain has a different physics, a different incumbent, and a different margin profile.

The Framework: Three domains, three physics, three markets

The single most useful mental model for AI networking is that there is no such thing as “the network.” There are three, nested like Russian dolls, and they are governed by different constraints. Scale-up is the innermost: it fuses a small number of accelerators into a single, coherent memory fabric where any GPU can read another’s memory in roughly 100 nanoseconds. Scale-out is the cluster fabric that connects those tightly-coupled islands into a machine of tens or hundreds of thousands of GPUs. Scale-across is the newest layer, connecting entire AI factories across buildings and metros as single sites run into power ceilings.

The reason this taxonomy matters commercially is that bandwidth, latency, and reach trade off against each other and against cost. Copper is nearly free and draws almost no power, but it dies at about two meters. Optics reach kilometers but burn watts and dollars per bit. The entire architecture of an AI supercomputer is an exercise in keeping the highest-bandwidth traffic on the cheapest, shortest, coolest medium — and pushing to light only when distance forces the issue. Follow the money, and it maps almost perfectly onto that physics.

Why It Gates Scaling: The utilization gap is a networking problem

The reason the interconnect commands this share of spend is visible in a single metric: Model FLOPS Utilization, the fraction of a GPU’s theoretical throughput a real training run actually achieves. Frontier runs commonly sustain under 50% MFU — more than half of the silicon you paid for is idle at any instant — and communication is a primary culprit. As models shard across more GPUs and parallelism widens, the share of wall-clock time spent moving data rather than computing rises. Past a certain scale the network, not the accelerator, sets the ceiling. That is the whole reason scale-up bandwidth (NVLink) and lossless scale-out fabrics exist: to keep the compute fed.

The proof is operational, not theoretical. When xAI built Colossus — 100,000 H100s, since expanded toward 200,000 GPUs including GB200 — on NVIDIA Spectrum-X, the fabric sustained 95% delivered throughput with zero flow-collision packet loss. NVIDIA pegs standard Ethernet at roughly 60% in the same role. That 60%→95% is a ~1.6× swing in effective bandwidth from the fabric alone — the difference between a GPU that earns and a GPU that waits. At a cluster costing tens of billions, that gap is the single highest-return line item in the building.

Scale-Up · The Memory Fabric: NVLink, NVSwitch, and the copper moat

Scale-up is where NVIDIA is strongest and where the physics are most punishing. When a model is sharded across GPUs using tensor parallelism, those GPUs exchange activations on every layer of every forward and backward pass. That traffic is enormous, latency-sensitive, and must look to software like shared memory rather than a network. The answer is NVLink: a memory-semantic interconnect that today delivers 1.8 TB/s per Blackwell GPU — roughly 14× a PCIe Gen5 link — and rises to 3.6 TB/s per GPU on the Rubin generation shipping into 2026.

The trajectory is a clean doubling per generation, and it is the tightest performance curve in the entire stack. In six years NVLink per-GPU bandwidth has gone up roughly 6×, from 600 GB/s on the A100 to 3.6 TB/s on Vera Rubin. That is not incidental; it is the moat. Competitors can match a GPU’s FLOPS far more easily than they can match a rack that moves a quarter of a petabyte per second internally.

How that bandwidth is delivered is as important as the number. Inside a GB200 NVL72, the 72 GPUs and their NVSwitches are wired through a copper backplane carrying roughly 5,000 cables and over two miles of copper, presenting every GPU to every other GPU in a single hop. NVIDIA chose copper deliberately: doing the same job in optics would have required transceivers and retimers drawing an estimated ~20 kW per rack just to light the NVLink spine. Copper is the moat and the power budget.

The strategic question is whether that moat stays proprietary. Two forces are pushing back. The first is UALink, an open scale-up standard ratified in April 2025 at 200 Gb/s per lane and scaling to 1,024 accelerators per pod, backed by AMD, Intel, Google, AWS, Microsoft, Astera Labs and 85+ members — conspicuously, everyone except NVIDIA. The second is NVIDIA’s own NVLink Fusion, which opens the NVLink fabric to third-party CPUs and custom silicon from Qualcomm, Fujitsu, Marvell and MediaTek. NVIDIA would rather rent its fabric to challengers than let them standardize around an alternative. That is a classic control-point defense.

Adjacent: the memory fabric (CXL)

A fourth fabric is forming alongside compute — for memory. CXL (read “What is CXL?”) lets a rack pool and share DRAM instead of stranding it in each server, attacking the “memory wall” that increasingly bounds inference. CXL 4.0, ratified in November 2025, carries up to 1.5 TB/s on bundled links; memory-pooling deployments target 2026–27, with early benchmarks showing ~20× gains for graph workloads and ~70% for recommendation models. It is the least mature domain — adoption is gated on orchestration and security, not silicon — but if memory disaggregation lands, it becomes a memory-semantic fabric with its own control points. Astera Labs’ Leo controllers are the merchant enabler to watch.

Scale-Out · The Fabric War: InfiniBand’s monopoly is breaking

Once a workload outgrows a single rack, it moves onto the scale-out fabric, and here the story of the last eighteen months is a genuine regime change. As recently as 2023, InfiniBand held roughly 80% of AI back-end networking. It earned that share honestly: InfiniBand is lossless by design, offloads collective operations into the switch via SHARP, and comes pre-integrated with NVIDIA’s stack. For frontier training, it remains the reference platform.

But Ethernet has flipped the market. In 2025, Ethernet more than doubled InfiniBand as the leading fabric for AI scale-out, and by Q4 2025 through Q1 2026 it accounted for roughly two-thirds of switch sales into AI clusters. IDC pegged the datacenter Ethernet switch market at $15.4B in Q1 2026, up 39.8% year-over-year, with NVIDIA — not Arista or Cisco — now ranked #1 by revenue on the strength of Spectrum-X. Amazon, Microsoft, Meta, Oracle and xAI are all building on Ethernet.

This shift has a precedent that should unsettle InfiniBand’s defenders: Ethernet has won this exact fight before, repeatedly. Token Ring, FDDI, ATM, and Fibre Channel each offered a real technical edge behind a proprietary or premium model — and each ceded the volume once Ethernet became merely “good enough” and far cheaper. Open, ubiquitous, and cost-declining beats fast-but-closed on a long enough timeline. InfiniBand is the current chapter, not an exception to the pattern.

InfiniBand’s most durable technical advantage is in-network computing. Collective operations like all-reduce — where every GPU must sum its gradients with every other GPU’s — are the heartbeat of distributed training, and they overwhelm a naively-built network. NVIDIA’s SHARP protocol moves the summation into the switch: instead of shuttling every gradient to every node, the switch aggregates as data flows through it. The result is roughly double the effective all-reduce bandwidth and up to 30% faster job completion on large runs. Ethernet’s answer — distributed, host-based collective libraries plus emerging in-switch offload — is closing the gap but is not yet at parity.

Topology · The Cost of Non-Blocking

Why the wiring diagram is a capital decision

A fabric’s logical shape matters as much as its silicon. Because training traffic is synchronized and all-to-all, any bottleneck anywhere stalls the collective, so large clusters are built as rail-optimized fat-trees: each GPU’s network port is mapped to a dedicated “rail,” so the same-index GPUs that must all-reduce together stay on a single leaf and avoid extra switch hops. This is the reference design at 16,000–100,000+ GPU scale across Meta, Microsoft, and NVIDIA’s own SuperPOD.

The expensive constraint is oversubscription. A training east-west path must be 1:1 non-blocking — any oversubscription stalls NCCL — which means the fabric has to carry every GPU’s full line rate at once. A non-blocking three-tier fat-tree roughly triples switch and optics count versus the oversubscribed networks enterprises tolerate, and that is precisely why networking is 10–15% of cluster cost rather than 2–3%. Topology is not an engineering detail; it is the mechanism that converts the physics of collectives into capital intensity.

The Silicon · Switches, NICs, DPUs

Where bandwidth is concentrated and offloaded

Underneath the protocol wars sits the silicon that actually forwards packets, and it is doubling on a metronome. The switch ASIC is the concentration point of the entire fabric, and Broadcom’s Tomahawk 6, shipping since June 2025, is the first to reach 102.4 Tbps on a single chip — double its predecessor, enough to wire a cluster of more than one million XPUs, with support for both 100G and 200G SerDes and optional co-packaged optics. The cadence is a clean 2× every two years, and it is the reason port speeds march from 800G today to 1.6T by 2027 and 3.2T by 2030.

This is where the merchant-versus-vertical split matters. Broadcom (Tomahawk, Jericho) and Marvell sell switch silicon to everyone; NVIDIA integrates Spectrum-X Ethernet and Quantum-X InfiniBand into its own systems. Broadcom’s combined AI revenue — custom accelerators plus networking silicon — ran $8.4B in Q1 FY26 (+106% YoY), $10.8B in Q2 (+143%), with guidance to ~$16B in Q3. The hyperscalers are deliberately funding a merchant alternative to keep NVIDIA honest.

At the edge of every server, the humble NIC has become a computer in its own right. NVIDIA’s ConnectX-9 SuperNIC delivers 800 Gb/s per port, and each Rubin GPU is paired with 1.6 Tb/s of scale-out bandwidth. But the more consequential shift is the rise of the DPU — a data-processing unit that offloads networking, storage, security and telemetry off the CPU and GPU entirely. NVIDIA’s BlueField-3 already does the work of roughly 300 CPU cores at ≤150W; BlueField-4, launching in 2026, adds 800 TOPS of on-die inference at 800 Gb/s. AMD’s Pensando Pollara 400 AI NIC and Salina DPU give the buyer coalition a credible second source.

Congestion Control · The Invisible Bottleneck

Why rated bandwidth and delivered bandwidth diverge

Here is the layer that separates a fabric’s spec sheet from its real performance. AI traffic is unlike anything Ethernet was designed for: it is synchronized, it moves in enormous “elephant” flows, and in a collective operation thousands of GPUs talk all-to-all at once. Two pathologies follow. Incast — many senders hitting one receiver at the same instant — overflows switch buffers and drops packets. And because classic Ethernet load-balancing hashes each flow onto a single path, a handful of elephant flows routinely collide on the same link, leaving it saturated while parallel links sit idle. In a collective, the slowest path sets the pace for all of them, so tail latency is job latency.

The fixes are the real intellectual property of modern AI networking. Lossless RoCEv2 uses ECN and priority flow control to signal congestion before buffers overflow. But the decisive move is packet spraying — per-packet load balancing that sprays a single flow across every available path — combined with adaptive routing and reorder-tolerant transport. NVIDIA’s Spectrum-X does this in hardware; Alibaba runs Oblivious Packet Spraying; and the Ultra Ethernet Consortium’s UET transport builds packet spraying and credit-based incast control into the standard. Done well, this lifts effective fabric utilization from roughly 50% toward 90%+ — a direct multiplier on the return of every GPU in the building.

Reliability · The Tail That Stalls the Job

At scale, the network’s job is to keep the run alive

There is a failure mode that never appears on a spec sheet yet dominates real economics. At scale, hardware breaks constantly, and synchronous training is brutally intolerant of it. Meta’s public Llama 3 405B run — 16,384 H100s over 54 days — logged 419 unexpected failures, roughly one every three hours, with about half tracing to GPUs, NVLink, or HBM memory. Because the job is synchronous, a single failed GPU or link can halt all 16,384.

Failures scale with component count. Hold the per-GPU rate constant and a 100,000-GPU cluster faces an interruption every ~30 minutes; the million-GPU clusters now being planned face hundreds per day. Meta still achieved >90% effective training time — but only through relentless checkpointing and a fabric that can detect a bad link and reroute around it in hardware. This is why link-level retry, adaptive routing, and fast fault isolation now matter as much as raw bandwidth. Reliability is the fabric’s third axis, alongside bandwidth and latency, and it is the one that gets harder every time a cluster doubles.

Inference · The Workload Inverts

Serving got more network-bound, not less

The single biggest shift in 2026 is that the workload changed. Inference passed half of all AI compute in 2025 and reaches roughly two-thirds in 2026. The lazy inference is that serving needs less network than training. The opposite is happening: modern inference is disaggregated and sparse, and both properties push traffic onto the fabric.

Two forces drive it. First, disaggregated serving splits the prefill and decode phases onto separate GPU pools, which means the multi-gigabyte KV cache must move between them at wire speed — NVIDIA open-sourced NIXL precisely for this, shuttling KV tensors from prefill to decode VRAM over RDMA, RoCE, or NVMe and cutting cost per token 3–5×. Second, mixture-of-experts models — now more than 60% of new open releases — route every token through a sparse all-to-all across experts; DeepSeek’s DeepEP saturates 400G NICs, and a full DeepSeek-R1 forward pass demands on the order of 13.7 TB/s of bandwidth. Inference has become a distributed-systems problem with the network at its center, which is why Dell’Oro projects scale-up Ethernet bandwidth to rise 15× in five years.

The market narrative says inference is cheap and local. The architecture says inference is a bandwidth problem wearing a latency mask.

Optical Networking · The Power Wall

Light is no longer the afterthought — it is the constraint

Every scale-out and scale-across link that exceeds a few meters is optical, which means optics now sit on the critical path of the entire AI build-out — on cost, on supply, and above all on power. A single pluggable transceiver draws about 15W at 800G and ~30W at 1.6T; a large GPU is fed by several of them, and at cluster scale optics consume a double-digit percentage of total AI-factory power. Conventional DSP-based pluggables burn roughly 21 picojoules per bit. When you are provisioning gigawatts, that is real money and real megawatts.

The industry’s response is a march to shorten the electrical path and delete power-hungry silicon. Linear pluggable optics (LPO) remove the DSP entirely, cutting module power 30–50% and shaving latency, and are projected to cover over 40% of short-reach 800G links. Co-packaged optics (CPO) go further, integrating the optical engines directly onto the switch package: NVIDIA claims 3.5× better power efficiency, 4× fewer lasers, and 10× the network resiliency. Broadcom’s Tomahawk 6 “Davisson” is already shipping 102.4 Tbps of optically-enabled switching, and NVIDIA’s Quantum-X Photonics InfiniBand reaches production in early 2026, with Spectrum-X Photonics Ethernet following in the second half.

A different bet: optical circuit switching

Not everyone is buying more packet switches. Google took a structurally different path with optical circuit switching (OCS) — MEMS mirror arrays that steer light from rack to rack without ever converting it to electrons. Its Jupiter fabric has used OCS (codenamed Palomar) across every TPU v4 and v5 pod, and the latest Ironwood generation pairs a 3D-torus with the Apollo OCS. The numbers are hard to argue with: a single OCS draws about 100W versus ~3,000W for an equivalent electrical packet switch — a ~95% reduction — contributing to 40% lower network power, 30% lower cost, and 50× less downtime. An OCS can’t inspect or buffer packets, so it fits the regular, reconfigurable topologies of TPU pods better than general GPU clusters. But it is the clearest existence proof that the packet-switched, convert-at-every-hop model is not the only way to build an AI fabric — and the OCP stood up a standardized OCS project in 2026.

The Money · Market Structure

Sizing the fabric — and who captures it

Pull back to the market map and the numbers are large and fast. Data-center capex is on track to approach $1 trillion in 2026 and hit $1.7 trillion by 2030. Within that, Dell’Oro sees AI back-end switching alone clearing $100B in annual spend by 2030. The optics attach is growing even faster: AI-cluster optical transceivers went from roughly $10B in 2024 to $16.5B in 2025 to a projected $26B in 2026 — ~60% growth two years running — with LightCounting’s upside scenario reaching $100B by 2030. The binding constraint is supply, not demand: InP laser and EML capacity is running roughly 30% short of orders.

The pace is set by a SerDes cadence that doubles lane rate every two-to-three years, dragging port speeds and optics with it. The majority of AI back-end ports have already moved to 800G; Dell’Oro expects 1.6T by 2027 and 3.2T by 2030. Each step is a forced upgrade cycle for switches, NICs, optics, and cabling — a recurring-revenue engine for whoever owns the reference design.

And who is capturing it? NVIDIA’s networking business alone is now running at roughly a $60B annualized rate — Q1 FY27 networking revenue hit $14.8B, up 199% year-over-year, with Spectrum-X past a $10B run-rate on its own. That single line item is larger than Broadcom’s entire AI semiconductor business and dwarfs Arista and Cisco’s AI franchises. The merchant challengers are growing fast — Arista’s AI revenue roughly doubles to $3.25B in 2026, Cisco’s AI infrastructure orders reach $9B — but the scoreboard shows how much of the value the platform owner still captures.

The megacap scoreboard understates where the sharpest returns sit. Beneath NVIDIA and Broadcom is a layer of specialists whose growth rates are, in percentage terms, even steeper — because they sell the connective tissue every architecture needs regardless of who wins the fabric war. Astera Labs (retimers, PCIe/CXL fabric switches, active cables) grew Q1 2026 revenue 93% YoY to $308M, a ~$1.2B run-rate, and now ships a 320-lane scale-up fabric switch. Credo, whose active electrical cables replaced flaky passive copper in hyperscale racks, roughly tripled fiscal-2026 revenue to $1.3B. Marvell’s data-center business — custom XPUs plus Inphi-derived optical DSPs — cleared $6B en route to a targeted $15B by FY28.

The scarcest tier — optical modules and their laser inputs — is where a set of mostly-Asian specialists captures the volume. InnoLight leads 800G at an estimated ~35% share; Eoptolink grew 175% to $1.2B at a 33% net margin; Coherent and Lumentum anchor the lasers and coherent DCI. The top five are roughly half the market. This layer is the cleanest expression of the thesis: pure connectivity exposure and growth that outruns the megacaps — with the offsetting risk that modules commoditize and CPO migrates value back to the switch vendor.

Make the stakes concrete. A fabric is not just a cost line; a better one converts directly into recovered compute. Drag the cluster size below — the bandwidth tax is what lifting delivered throughput from a vanilla ~60% to Spectrum-X’s ~95% is worth every year, alongside the optics bill and the failure interval you are signing up for.

The Investment Lens

Reading the stack as a set of control points

The layers of this stack are not equally defensible, and that distinction is the entire investment question. Sort it on the durable questions — structural advantage, capital intensity, margin durability, replacement risk — and a clear hierarchy of moats appears.

At the top sits the scale-up memory fabric: NVLink is the deepest moat in AI hardware because it is vertically integrated, copper-based, and physically hard to second-source. Just beneath it, and consistently underpriced by the market, is the congestion-control and transport software — adaptive routing, telemetry, and collective offload are what turn rated bandwidth into delivered bandwidth, and they travel with the platform. Switch silicon is durable but merchant, where Broadcom’s scale is the advantage. And optics — the fastest-growing layer — has the least durable margins: modules are commoditizing, the real scarcity is upstream in lasers and EML chips, and CPO is migrating value from the module maker to the switch vendor.

Start with the fabrics themselves. Every interconnect is a different point in a four-dimensional trade space — bandwidth, latency, reliability, and openness — and no fabric maximizes all four. NVLink and InfiniBand win the performance axes; Ethernet wins openness. On a five-year horizon, openness is the axis that has historically compounded.

Put probabilities on it. The one variable that moves the most enterprise value is NVIDIA’s share of AI networking revenue over the next three years — today roughly 65–70%. Base case, weighted at ~55%: Ethernet and open standards keep diffusing, NVIDIA’s platform bundling defends a slipping-but-dominant position, and share settles near half. The bull case (~25%) has NVLink Fusion, Spectrum-X and CPO holding share above 60% because open collectives and transport software lag. The bear case (~20%) has UALink and Ultra Ethernet maturing fast enough that hyperscalers standardize on open scale-up and merchant silicon, dragging share below 45%. Expected value lands around 52–54% — still dominant, but no longer a monopoly.

Bottom Line

The interconnect is the investable layer

The AI narrative has quietly become a networking narrative. Three nested fabrics — scale-up over copper, scale-out over light, scale-across over the metro — now determine whether a warehouse of accelerators behaves like one supercomputer or an expensive pile of idle silicon. The economics have followed: networking is a double-digit share of cluster cost, a ~$60B annual business for NVIDIA alone, and a market Dell’Oro sizes above $100B by 2030.

Own the bottlenecks — scale-up integration, transport software, switch-silicon scale, and upstream optical supply — and underwrite the open-standards coalition as the force that slowly redistributes the rest.

For investors, the signal is to price durability, not growth. The most defensible positions are the vertically-integrated scale-up fabric and the transport software that rides with it; the most durable merchant franchise is switch silicon at scale; and the scarcest physical bottleneck is upstream optical supply — lasers and EML chips — not the modules themselves. The commodity risk sits precisely where the growth headlines are loudest. The single highest-leverage variable over the next three years is whether the open coalition — UALink and Ultra Ethernet — can standardize the scale-up domain the way it already flipped scale-out. That contest, more than any FLOPS number, will decide where the margins in AI infrastructure ultimately settle.

Every AI product you have ever used is, mechanically, the same object: a token generator. Strip away the branding and a chatbot, a coding agent, a search summary, and an image caption are all the identical operation — a trained model run forward to predict the next token, then the next, then the next. That operation is inference, and in 2026 it stopped being a footnote to training and became the entire game. In May 2026 Google disclosed it was processing 3.2 quadrillion tokens per month across its surface area — roughly 38 quadrillion tokens annualized — a 7× increase from 480 trillion a month a year earlier, which was itself up from 9.7 trillion in early 2024. None of that is training. It is the cost of answering.

And yet inference is still treated as a black box: a prompt goes in, an answer comes out, a GPU is invoked somewhere in between. This piece opens the box. We take exactly one prompt and follow it literally — through fifteen physical and logical stops inside a datacenter — and explain precisely what each one does. By the end, “AI infrastructure” should stop being an abstraction and become something you can latency-budget, cost per token, and invest against.

The stakes behind that single round trip are the largest capital reallocation in the history of computing. Inference crossed roughly two-thirds of all AI compute in 2026, up from about one-third in 2023 and one-half in 2025. The four largest hyperscalers guided to ~$725B of 2026 capex, up 77% year over year, more than 60% of it going to power, cooling, and shells rather than chips. The market for inference-specific silicon alone will clear $50B in 2026. The single unit that ties this entire economy together is the token — and the token’s price is falling ~200× per year even as volume grows 7× per year. That scissor is the whole story: demand outruns a cost collapse, which is exactly the condition that turns an input into an economy.

The argument in five points

1. Inference is now roughly two-thirds of AI compute and 80–90% of a model’s lifetime cost. The token, not the model, is the unit of economic output.

2. Two phases, two bottlenecks: prefill is compute-bound (sets time-to-first-token) and decode is memory-bandwidth-bound (sets speed and cost). Nearly every optimization targets one of them.

3. Unit cost is falling ~200×/year while volume grows ~7×/year — Jevons, not deflation. “Enormous” and “high-margin” are different claims.

4. Durable value sits at the physical bottlenecks — memory bandwidth, scale-up interconnect, optics, and power. The inference-software layer is a real but contested business.

5. Buying inference is a weighted trade — latency (TTFT + speed + p99), blended cost, reliability (including the functional-availability gap), security, deployment, model coverage, and portability — scored against your workload.

The machine the economy runs on

The reason to care about the plumbing is economics. Training a model is a capital event — a one-time burn that produces a fixed asset. Inference is the cost of goods sold: it recurs on every query, forever, and it scales linearly with usage. The industry rule of thumb is that inference eventually accounts for 80–90% of the lifetime compute cost of a deployed model. So the shift in Figure 2 — inference passing from a third of compute in 2023 to two-thirds in 2026 — is not a fashion. It is the system reaching steady state, where most of the world’s accelerators spend most of their time serving, not learning.

Two forces make this the defining infrastructure market of the decade. The first is that the frontier moved from bigger models to longer thinking: reasoning and agentic systems spend 10–100× more tokens per query. DeepSeek-R1 matched OpenAI’s o1 while emitting an order of magnitude more tokens per answer; agentic workflows — with their retries, tool calls, and context reloads — run 5–25× more expensive per task than a single-shot completion. The second force is a price collapse without modern precedent: the cost to serve a fixed quality level has fallen at a median ~200× per year since early 2024. Cheaper tokens do not reduce spend; they unlock new workloads faster than price falls. That is the Jevons paradox, and Figure 3 is what it looks like.

So the rest of this article is a single claim, made concrete: every technique below — batching, paging, quantization, speculation, disaggregation, and the network fabric itself — exists to move one number, cost per token at a target latency. Follow the specimen prompt and you will see exactly where each fraction of a cent and each millisecond is won or lost. Fifteen stops. Here is the first.

Step 1: User — the prompt becomes numbers

The journey begins before the datacenter. When you press enter, your client sends text over HTTPS — but the model never sees text. It sees integers. A tokenizer (a byte-level byte-pair encoder) chops the string into sub-word tokens, each mapped to an ID drawn from a vocabulary of roughly 100k–200k entries. Our 12,000-token RFP plus its question becomes a flat sequence of 12,022 integer IDs. Tokenization is deterministic, CPU-side, and effectively free — but it defines the bill. You are charged per token in and per token out, and the model’s context window (200k–1M tokens on 2026 frontier models) is a hard ceiling on how much of that RFP can enter at all.

Step 2: API Gateway — the front door

The request — still just an HTTPS body carrying those IDs — hits the API gateway, the datacenter’s front door. It terminates TLS, parses and validates the request against a schema, enforces API versioning, applies coarse rate limits, attaches a trace ID, and emits the first metering event. It is deliberately dumb and brutally fast: no model logic, just admission and bookkeeping at millions of requests per second, typically on Envoy- or NGINX-class proxies fronted by a web-application firewall. Latency budget: sub-millisecond. Its job is to reject the 5% of traffic that is malformed, over-quota, or hostile before any expensive resource is touched.

Step 3: Authentication — identity, tier, and the meter

Before a single GPU is touched, the request must be attributed. Authentication resolves the API key or OAuth token to an organization, checks its rate-limit tier and spend caps, and decides billing — the per-token price that will apply, whether the caller is eligible for cached-input discounts (a 50–90% reduction we will meet at stop 7), priority lanes, or cheaper batch pricing. It is also the security boundary: org-level data isolation and abuse defense. This stop is where a request stops being anonymous bytes and becomes a metered, priced, and isolated unit of work.

Step 4: Load Balancer — placing the request

Now the work must be placed on hardware. A conventional load balancer spreads requests across many identical model replicas using health checks and live load signals. But for LLM serving, naive round-robin is actively wasteful: two requests that share a long prefix — the same system prompt, the same pasted RFP — should ideally land on the same replica so the expensive prefix computation can be reused from cache. So modern AI load balancing is becoming cache-aware, which is exactly why it blurs into the inference router at the next stop. The load balancer also enforces backpressure: when the fleet is saturated, it is the component that decides whether you queue or get a 429.

Step 5: Inference Router — model, silicon, and kernel selection

The router is where inference gets interesting, and where a startling amount of gross margin is won or lost. In a few milliseconds it answers three questions. Which model? A frontier 70B, a distilled 8B, a reasoning variant, or a speculative draft + target pair — serving a query on a 70B when an 8B meets the service-level objective is pure margin destruction. Which silicon? This is hardware selection, and it is phase-dependent: the compute-bound prefill wants raw FLOPs, while the memory-bound decode wants HBM bandwidth, so the “right” GPU differs within a single request. Which replica? Cache-aware routing sends the request to the instance that already holds this prefix’s KV cache, converting a 12,000-token prefill into a near-free cache hit.

This stop is also home to the compiler / autotuner layer — the systems that, ahead of time, compile and tune the exact GPU kernels for a given model, shape, and chip, and at runtime select the cheapest execution plan. This is not academic: it is a business. Together AI, Fireworks, Baseten, and Modular were each built, in large part, at this layer — turning “which kernel on which chip at which precision” into a product, then monetizing it with their own inference infrastructure. We return to that model, and its valuations, at the end of the journey.

Step 6: Scheduler — the batching engine that funds it all

If a single component makes GPU economics work, it is the scheduler. A lone decode step underuses a GPU catastrophically: at batch size 1, an H100 runs at roughly 30–40% streaming-multiprocessor utilization because it is starved on memory, not math. The scheduler’s answer is continuous (iteration-level) batching: rather than fixed batches, it interleaves many sequences and, at every forward pass, admits freshly-arrived requests and evicts finished ones, keeping the GPU saturated pass after pass. This one trick is the core of vLLM, which delivers 2–4× the throughput of prior serving systems and 3–5× the traffic of a naive PyTorch loop on the same H100.

Step 7: KV Cache — the memory that makes generation possible

Now the request is on a GPU, and we hit the single most important data structure in inference. Inside every attention layer, each token produces a Key and a Value vector. To generate token N+1, the model must attend to the K and V of every prior token — so recomputing them at each step would cost O(n²) and be ruinous at a 12,000-token context. Instead they are cached: the KV cache. Prefill computes and stores K/V for all 12,022 positions once; each subsequent decode step appends one token’s K/V and reads the rest. The KV cache is the largest and most dynamic consumer of GPU memory during serving, and managing it well is most of what a modern inference engine does.

The hard part is fragmentation. Sequences grow to unpredictable lengths, and classic contiguous allocation strands 60–80% of memory. vLLM’s PagedAttention borrows the operating-system trick of virtual memory: the KV cache is split into fixed-size blocks (“pages”), allocated on demand and mapped through a page table, so physical blocks need not be contiguous and finished sequences free their pages instantly. That is the mechanism behind vLLM’s 2–4× throughput — and it is what lets many users’ sequences safely share one GPU.

That sharing is not just a memory optimization — it is a price. If many requests carry the same prefix (a system prompt, a long few-shot preamble, our pasted RFP), its KV cache can be computed once and reused across calls. Providers now sell this directly as prompt / prefix caching: Anthropic charges cached reads at 0.1× input ($0.30 vs $3.00 per million tokens); OpenAI’s GPT-5.x line prices cached input at $0.50 vs $5.00 — both a 90% discount, with roughly 85% lower latency on long prompts. For our specimen, caching the RFP collapses a 12,022-token prefill into a near-free cache hit on every follow-up question.

Step 8: GPU Memory — the real constraint on who you can serve

A serving GPU’s on-package memory holds three things at once: the model weights (fixed), the KV cache (grows with the number of concurrent sequences and their length), and activations and workspace (transient). The weights are the floor; whatever remains is the KV budget, and that budget — not compute — usually caps concurrency. A 70B model is ~140 GB in FP16 and needs two 80 GB H100s just to load; in FP8 it is ~70 GB and fits one GPU with room left for KV cache. This is the first place quantization pays for itself: every gigabyte shaved off the weights becomes KV headroom, and KV headroom is the number of users you can serve simultaneously.

Step 9: HBM — the bandwidth wall and the roofline

This is the counterintuitive heart of the whole machine. Prefill and decode stress entirely different resources. Prefill processes all 12,022 input tokens in parallel — one enormous matrix multiply that saturates the tensor cores. It is compute-bound, and it sets TTFT. Decode is the opposite: to produce each single next token, the GPU must stream the model’s entire weight tensor (plus the growing KV cache) out of HBM, once per token. That read time is fixed by memory bandwidth, not math. At batch size 1, decode’s arithmetic intensity is about 1 FLOP per byte — far below the roofline’s ridge point of ~410–590 FLOP/byte — so the tensor cores sit almost entirely idle, waiting on memory.

So the industry is marching down the precision ladder: FP16 → FP8 → FP4. Fewer bits per weight means fewer bytes to read per token, which directly lifts the memory-bound decode ceiling. NVIDIA’s NVFP4 — a 4-bit floating-point format native to Blackwell’s 5th-generation tensor cores — delivers roughly 2–3× the arithmetic throughput of FP8 and ~1.8× memory reduction while holding within about 1% of baseline accuracy, contributing to end-to-end inference speedups up to 5× versus Hopper. Every bit dropped is bandwidth reclaimed and a token made cheaper — the single most direct lever on cost of goods sold in the building.

Prefill is compute-bound and sets your time-to-first-token; decode is memory-bandwidth-bound and sets your speed and cost. Nearly every technique in inference is an attack on that decode memory wall — batch bigger, quantize smaller, speculate ahead. Read: Why KV Cache and Memory Drive AI Economics.

Step 10: CUDA Kernels — where tokens become arithmetic

The execution plan is now math, and math on a GPU runs as kernels — small programs executed across thousands of cores. The naive way to run a transformer is hundreds of separate kernel launches, each reading its inputs from HBM, computing, and writing results back. For a bandwidth-bound workload that round-tripping is fatal. So the entire craft is kernel fusion: collapse many operations into one kernel that keeps intermediate data in fast on-chip SRAM and touches slow HBM as little as possible. The canonical example is FlashAttention, which computes attention in tiles with an online softmax, cutting HBM reads and writes from quadratic to linear in sequence length for 2–4× speedups. FlashAttention-3 exploits Hopper’s asynchronous engines and FP8 to reach 840 TFLOPS on an H100 — about 85% of peak.

One kernel-level trick deserves its own spotlight because it is the highest-ROI optimization in single-stream serving: speculative decoding. Since decode is memory-bound — the GPU reads all the weights per step no matter what — you can verify several candidate tokens for almost the same cost as generating one. A small, cheap draft model proposes the next K tokens; the large target model checks them all in a single parallel pass and accepts the longest correct prefix. The output is mathematically identical to ordinary decoding, but 2–4× faster. Modern methods like EAGLE-3 push draft acceptance rates above 75%.

Step 11: NVLink — when one GPU is not enough

Our 70B specimen fits on a single GPU, but the frontier does not: trillion-parameter and mixture-of-experts models are sharded across many GPUs, which means the GPUs must talk to each other — on every layer, for every token. This is the point people miss: networking sits directly inside the decode loop, not off to the side. The first fabric is scale-up — NVLink — the ultra-fast interconnect that fuses many GPUs inside a rack into one logical accelerator. NVLink 5 moves 1.8 TB/s per GPU, about 14× a PCIe Gen5 link. The GB200 NVL72 wires 72 Blackwell GPUs and 36 Grace CPUs into a single NVLink domain with 130 TB/s aggregate bandwidth and 13.4 TB of unified memory — one rack behaving as one machine, delivering up to 30× the inference throughput of an H100 cluster on trillion-parameter models, while drawing ~120 kW.

Why is such extreme bandwidth necessary? Because parallelism creates traffic. Tensor parallelism splits each layer’s matmul across GPUs, and after every one the partial results must be summed with an all-reduce collective — many times per token. Worse, mixture-of-experts models route each token to a handful of experts scattered across GPUs, triggering all-to-all communication that frequently becomes the dominant inference bottleneck. DeepSeek’s production stack uses eight 400 Gbps NICs per node and a bespoke library (DeepEP) to overlap this exchange with computation so the GPUs never stall.

Step 12: Switch — the crossbar and the congestion problem

A fabric needs switches. Inside the scale-up domain, NVSwitch is the crossbar: 144 NVLink ports and 14.4 TB/s of non-blocking switching, so all 72 GPUs can talk at full speed at once. Beyond the rack, the scale-out network uses its own switches — NVIDIA’s Quantum-X800 (InfiniBand) and Spectrum-X800 (Ethernet), or Broadcom’s Tomahawk 6 at 102.4 Tbps (64 × 1.6T ports). Two things make an AI switch unlike a cloud switch. First, rail-optimized topology: GPUs are cabled so that the n-th GPU in every server homes to the same “rail” switch, minimizing hops for collectives. Second, in-network computing and congestion control: switches can perform reductions in-fabric (SHARP), and they must absorb the synchronized “incast” of thousands of GPUs finishing a step at the same instant. One congested link stalls an entire collective — the tail-latency problem — so adaptive routing is existential, not optional.

Step 13: NIC — the GPU’s on-ramp to the cluster

A packet leaving the GPU for another rack goes through a network interface card — in 2026, almost always a SmartNIC or DPU (NVIDIA BlueField and peers). Its core job is RDMA over Converged Ethernet (RoCE) or InfiniBand verbs: letting a remote GPU read this GPU’s memory directly, bypassing both CPUs, at 400 Gb/s today and 800 Gb/s becoming standard. But the DPU is a full programmable computer on the NIC: it offloads congestion control, encryption, storage virtualization, and multi-tenant isolation from the host CPU. In a rail-optimized cluster each GPU typically owns a dedicated NIC — DeepSeek pairs eight GPUs with eight 400 Gb/s NICs — so no accelerator shares an on-ramp. This is where “the network” physically begins for a byte, and it is the boundary where our data-gravity theme becomes literal.

Step 14: Ethernet — the scale-out fabric, and the optics beneath it

Historically AI clusters ran on InfiniBand for its ~1–2 µs latency and lossless fabric; Ethernet with RoCEv2 sat at ~5–10 µs and was treated as second-tier for training. That hierarchy is inverting. The Ultra Ethernet Consortium shipped UEC 1.0 in June 2025 — a 560-plus-page rebuild of the Ethernet stack for AI — closing the latency and reliability gap, and Dell’Oro projects Ethernet overtakes InfiniBand in AI back-end networks by 2027. 800G is already the default port speed, with Broadcom’s Tomahawk 6 pushing 102.4 Tbps. For inference specifically — cost-sensitive, multi-tenant, enterprise-adjacent — Ethernet’s economics and open ecosystem are decisive.

Underneath the fabric is its least-discussed cost center: optics. Optical transceivers are roughly 60% of networking cost and 45% of networking power; with networking at ~15–18% of total cluster cost, optics alone are close to 10% of the entire bill. A single 800G pluggable can burn more than 500 W across a switch — more than the switching ASIC itself. The AI optical-transceiver market is running from ~$16.5B in 2025 to ~$26B in 2026 (+57% YoY). The structural fix is co-packaged optics: move the optics onto the switch package, cutting a 1.6T link from ~30 W to ~9 W, with NVIDIA claiming 5× power efficiency and 10× resiliency for its photonics switches shipping in 2H 2026.

For any model too big for one GPU, the network sits inside the decode loop — it runs on every layer, every token. Managing the ~20–40× bandwidth cliff between scale-up (NVLink) and scale-out (Ethernet) is the whole game.

Step 15: Response — detokenize, stream, and settle the bill

The last token is generated, and now the trip runs in reverse. Detokenization maps the integer IDs back into text, and the answer streams token-by-token back out through the NIC, the switch, the load balancer, and the gateway to the user — typically over server-sent events, so tokens appear as they are produced. This is why time-to-first-token dominates perceived speed: the user starts reading at ~0.3 s even though the full answer takes seconds to finish. The final act is the meter closing the bill — input tokens (possibly cached), output tokens (the expensive ones), priced at the tier that authentication assigned back at stop 3. Fifteen stops, one answer.

Final Step: Putting it together: the inference OS

Walk the fifteen stops again and a pattern emerges: the hard problems — batching, KV paging, prefill vs decode, hardware selection, collective communication — are all really one problem, which is scheduling heterogeneous work across a memory-and-network hierarchy to keep expensive silicon saturated. The 2026 answer is disaggregated serving: split the compute-bound prefill and the memory-bound decode onto separate GPU pools, each scaled and tuned independently, and stream the KV cache between them. This is now standard across NVIDIA Dynamo, vLLM, SGLang, llm-d, and Mooncake.

The business of the stack: compilers that became clouds

The most interesting companies on this map sit at stops 5 and 10 — the compiler, kernel, and autotuner layer — and they discovered the same thing: the way to monetize a better kernel is not to license it, but to run it. Own the inference infrastructure, sell tokens, and keep the efficiency delta as gross margin. The logic is airtight. Every 2× you win in kernels, batching, quantization, or speculative decoding is 2× more tokens per GPU — and if you are the one operating the GPUs, that is margin you capture rather than a benchmark you publish. An autotuner that compiles the cheapest kernel for every model × shape × chip is, in effect, a money printer bolted to a fleet.

The growth rates are the argument. Baseten’s run-rate went from ~$200M (Dec 2025) to ~$600M by March 2026 — roughly 1,900% YoY — as it closed a $1.5B round at $11–13B, up from $5B five months earlier. And Modular’s ~$3.9B sale to Qualcomm is the clearest signal of all: a chipmaker buying a hardware-agnostic compiler to attack NVIDIA exactly where the moat is thinnest.

Choosing an inference provider: the real decision criteria

If you are buying inference rather than building it, the fifteen-stop machine collapses into a purchasing decision — and the most common mistake is optimizing for a headline price per million tokens. There is no best provider in the abstract; there is only the best provider for a workload shape. A real-time voice agent, a nightly batch summarizer, a multi-step autonomous agent, and a HIPAA-bound enterprise app weight the same criteria completely differently. The framework below is the one that survives contact with production.

The three everyone names — measured correctly

Latency is two numbers, not one. Time-to-first-token (the prefill latency) governs how fast the answer starts; inter-token latency, or output speed, governs how fast it finishes. A provider can win one and lose the other, and the number that matters for user experience is the p99 tail, not the median — a great p50 with a poor p99 still produces visible stalls. Typical GPU inference lands first tokens in ~400–600 ms; the speed-silicon clouds compress that sharply, with Groq and Cerebras posting sub-100–150 ms TTFT and output speeds of 1,600–2,100+ tokens/second on Llama-70B-class models — roughly 4–6× a typical GPU stack.

Cost is blended, not posted. The list price is two numbers — input and output per million tokens — and output is priced ~4–5× input and usually dominates. Your real unit cost depends on your input:output ratio, your cache-hit rate (prompt caching cuts input 50–90%), and whether the work can use a batch tier (often ~50% cheaper). Posted rates in mid-2026 span two orders of magnitude — from roughly $0.04–$0.20 per million on cost-optimized open-model endpoints (DeepInfra; Groq at $0.15/$0.60) to several dollars per million for frontier models — so two providers with identical headline prices can differ 3× on your traffic.

Reliability is more than an uptime percentage. The formal SLA (Azure OpenAI publishes 99.9% for token generation; enterprises increasingly demand a latency SLA such as sub-200 ms TTFT for 99.99% of calls) is necessary but not sufficient. The failure modes that actually break products sit outside it: refusal spikes, silent model-version bumps that regress your evals, and quota-driven throttling under load. A vendor can honor 99.95% uptime and still degrade your application through any of these — which is why the mature answer is to pin model versions, negotiate capacity, and monitor functional availability yourself.

The seven everyone forgets

Beyond latency, cost, and reliability, seven criteria decide production fit. Throughput & rate limits (tokens-per-minute ceilings and burst headroom) gate agentic fan-out and scale. Security & compliance — SOC 2 Type II, HIPAA, ISO 27001, GDPR — is table stakes in regulated industries and slow to retrofit. Data residency & private deployment (zero-retention guarantees, VPC/BYOC, on-prem) is what clears enterprise procurement; most consumer-grade endpoints leave retention vague and offer no private path, quietly disqualifying themselves. Determinism & version control — fixed seeds, pinned checkpoints — protects your evaluation suite from silent drift. Model coverage & freshness (breadth, day-zero support for new open weights, fine-tune/LoRA hosting) sets how fast you adopt the frontier. Deployment flexibility (serverless vs dedicated vs self-host) sets your cost/control trade. And portability — an OpenAI-compatible API and clean multi-provider routing — is your insurance against every one of the above going wrong.

Pick your workload column first, then shop. A headline price per million tokens is close to meaningless without your input:output ratio, your cache-hit rate, and a latency SLA that covers the p99 tail.

Implications: where the value accruescore

Three conclusions follow. First, the physical bottlenecks compound: HBM bandwidth sets the decode ceiling, the NVLink scale-up domain is genuinely proprietary, and optics plus power are the scarce inputs — more than 60% of that $725B capex is already power and buildings, which makes tokens per watt the terminal metric. Second, the network is bifurcating: scale-up stays closed and defensible while scale-out goes open (Ethernet/UEC) and commoditizes, so the edge is the scale-up domain, optics/CPO, and congestion IP — not commodity switching. Third, the inference-software layer is real but contested: margin equals efficiency delta × utilization × scale, durable only where a player has converted speed into distribution and switching costs before the price collapse makes raw performance table stakes.

The decisive read: through 2027, back the bottlenecks — memory and bandwidth, scale-up interconnect, optics and power — and the inference platforms that have already turned efficiency into lock-in. Be skeptical of anything whose entire pitch is raw speed, because raw speed is exactly what the compiler layer, and NVIDIA’s own free tooling, are busy commoditizing. The token economy will be enormous — from 9.7 trillion to 3.2 quadrillion Google tokens a month in two years is not a curve that reverses — but enormous and high-margin are different claims, and the fifteen stops are where that difference is decided.

Bottom line: Back the physical bottlenecks — memory bandwidth, scale-up interconnect, optics, power — and the platforms that have turned efficiency into lock-in. Be skeptical of any pitch whose only moat is being faster; that is exactly what the compiler layer and NVIDIA's own free tooling commoditize.

Follow one token and the whole economy becomes legible.

Sources & further reading — accessed June 2026

1. Deloitte — More compute for AI, not less (inference ~2/3 of compute, 2026)

2. MarketsandMarkets — AI Inference Market size & forecast ($106B→$255B)

3. Tom’s Hardware — Big Tech AI capex reaches $725B in 2026

4. Introl — Hyperscaler capex & AI infrastructure

5. Shacknews — Google processing 3.2 quadrillion tokens/month

6. Crypto Briefing — Google 3.2Q tokens/month, 7× YoY

7. Google — Sundar Pichai, I/O 2026 keynote

8. Epoch AI — LLM inference price trends

9. TokenCost — AI Price Index (300× drop, 2023–26)

10. NVIDIA / SEC — Q1 FY2026 results ($39.1B DC)

11. Intellectia — NVIDIA record $81.6B quarter (May 2026)

12. Kwon et al. — PagedAttention / vLLM (arXiv)

13. Spheron — Continuous batching, PagedAttention, chunked prefill

14. RunPod — vLLM PagedAttention & continuous batching

15. Hao AI Lab (UCSD) — DistServe: prefill–decode disaggregation

16. Hao AI Lab — Disaggregated inference, 18 months later

17. NVIDIA — Introducing Dynamo (30× on DeepSeek-R1)

18. NVIDIA — Inference cluster economics & AI-factory tokens/s

19. Local AI Master — Speculative decoding: EAGLE, Medusa

20. Spheron — EAGLE-3 speculative decoding (2–4×)

21. NVIDIA — Introducing NVFP4 low-precision inference

22. Edge AI + Vision — Blackwell: impact of NVFP4

23. EmergentMind — Triton & CUDA kernels; FlashAttention-3

24. Spheron — HBM3e/HBM4 bandwidth & roofline for inference

25. Samanvya Tripathi — Prompt caching: 90% cost cut

26. PromptHub — Prompt caching across OpenAI/Anthropic/Google

27. NVIDIA — NVLink & NVSwitch (1.8 TB/s, 14.4 TB/s)

28. NVIDIA — GB200 NVL72 (72 GPUs, 130 TB/s)

29. Introl — NVLink & scale-up networking

30. Ultra Ethernet Consortium — UEC 1.0 launch

31. HPCwire — Ultra Ethernet has arrived

32. TrendForce — InfiniBand vs Ethernet scale-out

33. SemiAnalysis — Co-packaged optics: scaling with light

34. NVIDIA — Silicon photonics for agentic AI

35. TrendForce — AI optical transceiver market to $26B (2026)

36. DeepSeek — DeepEP expert-parallel comms library

37. Introl — Mixture-of-experts infrastructure

38. Introl — Inference-time scaling & reasoning models

39. Sacra — Fireworks AI revenue & valuation

40. SiliconANGLE — Fireworks $250M Series C at $4B

41. Startup Fortune — Baseten $1.5B round at $11–13B

42. Contrary Research — Baseten company profile

43. Sacra — Modular valuation & funding

44. Startup Fortune — Qualcomm to buy Modular (~$3.9B)

Underneath the anecdotes sit the numbers. Claude Code now authors more than 2.6 million commits a week — more than every other AI coding tool on earth combined — and that counts only public GitHub; the enterprise majority of its work runs in private repositories the tally never sees. Inside Anthropic, Claude already writes north of 80% of the code that ships to production, up from low single digits when the tool launched. Nine months after release it became the fastest software product in history to a $1 billion run-rate — and it is now several multiples beyond that, still accelerating.

The lazy reading is “the models got good.” The useful reading is structural. Of everything we ask machines to do, coding fell first — and the reason has little to do with how hard programming is and everything to do with grading. Software is the one discipline that ships with its own referee: a compiler that rules valid or invalid, a test suite that rules correct or incorrect, instantly and for free. That objective signal is the single scarcest ingredient in modern AI training — and almost no other kind of work supplies it. Claude Code is what happens when a frontier lab aims reinforcement learning at the one domain where the answer key already exists, then builds a harness good enough to act on a real computer. The rest of the economy is now a question of who can write the next answer key.

Why code was the first domino

Modern language models are pre-trained on text, then post-trained to be useful. The decisive lever in post-training is reinforcement learning: the model proposes, an environment scores the proposal, and the scores reshape the weights. The bottleneck is the scorer. For most knowledge work — write a strategy memo, summarize a deposition, design a campaign — there is no cheap, objective function that returns right or wrong. You need humans, and humans are slow, expensive, and inconsistent at scale.

A single Claude Code turn: It reads the repo, edits the source, runs the test suite, and only ships when the suite goes green — the same read → act → execute → verify loop it was trained on:

Code is the exception. Correctness is executable. You can run the program, run its tests, and read an unambiguous pass or fail — millions of times, in parallel, for the marginal cost of compute. This is the substrate behind the term of art reinforcement learning with verifiable rewards (RLVR): rather than training a model to match human preference, you train it against ground truth a machine can check. Code, math, and formal proofs are the domains where that ground truth is effectively free. Software has the largest and richest supply of it — every repository on earth is a corpus of problems with graders already attached.

How the loop is actually built

The mechanics are a loop, not a single forward pass. Start from a real task — a GitHub issue against a real repository at a known commit. The policy (the model) is dropped into a sandbox and samples a rollout: it reads files, greps the codebase, edits source, and runs commands across many turns. When it judges itself done, the harness runs the repository’s test suite. Tests pass, reward is 1; tests fail, reward is 0. That scalar — or a denser version that credits compilation, partial progress, and fault localization — is fed back through a policy-gradient algorithm (PPO- or GRPO-style) that shifts the weights toward the behaviors that earned reward. Repeat across millions of tasks and the model stops emitting plausible-looking code and starts producing code that runs.

Two properties make this potent. First, no human sits in the inner loop, so the data engine scales with compute rather than headcount — the defining economic feature of the approach. Second, the reward is grounded in execution, not in what looks correct, which is why benchmarks like SWE-bench Verified — drawn from real, resolved GitHub issues — became the industry’s yardstick. The failure mode is reward hacking: a model that learns to overwrite the test instead of fixing the bug. Anthropic has said it cut reward hacking in the Claude 4 generation specifically by improving environments, sharpening reward signals, and monitoring with prompted classifiers and execution checks that flag tampered tests. Clean reward design, not raw scale, is the hard part.

The harness is the product

A model that passes tests in a sandbox is necessary but not sufficient; it still has to operate a real computer. Anthropic’s second insight was that the interface between agent and machine — what they call the Agent-Computer Interface (ACI), deliberately analogous to the human-computer interface — is a first-class engineering surface. In building their SWE-bench agent, the team reported spending more time optimizing tools than the prompt. Rewriting tool descriptions so the model could call them reliably cut task-completion time by roughly 40%.

The philosophy is primitives over integrations. Claude Code ships a deliberately small toolset — read a file, list a directory, grep, make surgical edits, run a Bash shell — and lets the model compose them, rather than wrapping every external system in a bespoke connector. The same instinct produced the Model Context Protocol (MCP), Anthropic’s late-2024 open standard for plugging tools and data into agents, now broadly adopted across the industry. The strategic point compounds: the model is the engine, but the harness is what converts raw capability into reliable work — and it is far harder to clone than a benchmark score.

That moat extends one layer upstream, into the environments themselves. The harder it becomes to verify long-horizon work, the more the binding constraint shifts from the model to whoever can build and grade the training environments — the argument I make in RL Environments for Agentic AI, where verification, not raw model capability, is the real prize. Anthropic is already the single largest buyer of coding and computer-use environments — on the order of tens of millions of dollars a year, and growing an estimated 3–5x into 2026. It is not only training on verifiable rewards; it is buying up the supply of verification itself.

The model is the engine. The harness is the moat. One converges across labs; the other is years of accumulated environment and tool design.

Crossing the human-competitive threshold

The flywheel shows up directly in the scores. On SWE-bench Verified, Claude moved from 49% with 3.5 Sonnet in late 2024 to 80.9% with Opus 4.5 in November 2025 — the first model past 80%, a level above the human-competitive bar on the benchmark — and to 88.6% with Opus 4.8. The gains arrived with falling cost: Opus 4.5 matched the prior Sonnet’s best result using roughly three-quarters fewer output tokens. Just as important is task horizon. METR’s tracking shows the length of tasks agents can complete autonomously doubling every four to seven months, and accelerating. Each doubling moves the frontier from autocompleting a snippet, to refactoring a module, to owning a multi-day migration — the exact shift that turns a coding assistant into a coding agent.

The second-order effects are louder than the benchmark. Inside Anthropic, the share of production code written by Claude went from low single digits at launch to more than 80% by May 2026; the typical engineer now merges roughly 8x more code per day than in 2024, and on the hardest, spec-light problems — where no clean test exists to begin with — Claude’s success rate jumped about 50 points in six months, to 76%. In one April 2026 stretch, Claude shipped 800-plus fixes that cut a single class of API errors by a factor of 1,000. The recursive part is the part to sit with: Claude Code now writes most of the code for the next version of Claude. The loop that cracked coding is increasingly pointed at itself.

The fastest revenue curve software has produced

Capability converted into the steepest revenue curve software has produced. Claude Code reached a $1 billion run-rate six months after its May 2025 launch — faster than any commercial software product before it, versus roughly eleven months for ChatGPT and years for the canonical SaaS breakouts — then doubled to $2.5 billion three months later. It is the engine under Anthropic’s own ascent: company run-rate went from ~$87 million entering 2024 to $1 billion by year-end, $9 billion exiting 2025, and run-rate tracking past $55 billion by late June 2026 (above the $50B-plus Anthropic guided). The valuation kept pace — a $380 billion Series G in February became a confidential IPO filing at ~$965 billion on June 1, after a $65 billion Series H, with the company on pace for its first profitable quarter. Coding is the throughline: Anthropic holds an estimated 54% of coding-specific LLM spend, enterprise is now more than half of Claude Code revenue with business seats up roughly 4x year-to-date, and the base is contracted at the top — 8 of the Fortune 10 and ~70% of the Fortune 100 are customers, with Accenture alone training 30,000 staff on Claude.

What Claude Code is actually running at now

Anthropic’s last public figure — $2.5B, in February — is four months stale in the fastest-moving product in software. Three independent methods say the truth is multiples higher.

By cadence: Claude Code went $1B → $2.5B in the three months to February — 2.5x in a quarter. Even decelerating hard, to ~25% a month, that lands near $7–8B by late June. By revenue share: Claude Code was ~18% of company run-rate in February ($2.5B of $14B); with company run-rate tracking past $55B by late June and coding the lead driver, a steady 15–18% share is $8–10B. By third-party forecast: FutureSearch models Claude Code near $21B by May 2027 — a path that runs through roughly $8–10B today. The three converge on a best estimate of ~$8–9 billion in annualized run-rate as of mid-2026 — about 3.4x the last disclosed number in four months — with a bull case above $12B and a year-end exit in the $15–20B range. Even the floor of that band makes Claude Code, by itself, one of the five largest developer-software businesses on earth, built in thirteen months. The $2.5B headline you still see quoted understates the product by more than half.

Methodology — how we get to ~$8.5B
Two disclosed anchors ($1B Nov '25, $2.5B Feb '26), then three methods — cadence (~$8B), revenue share (~$8–10B), independent forecast (~$8–10B) — converging on ~$8–9B.

Past the entire pre-AI developer-tools market

Set Claude Code against the franchises that defined developer tooling before AI. GitHub, after sixteen years and a Microsoft acquisition, runs about $2 billion a year. GitLab does roughly $580 million; JetBrains, around $250 million. The entire standalone developer-tools software market was worth near $5–6 billion in 2024. Claude Code — which did not exist commercially before mid-2025 — was already past $2.5 billion at its last disclosure and, on the estimate above, is running at ~$8–9 billion today: more than GitHub, GitLab, and JetBrains combined, and larger than the whole pre-AI category by itself. The AI-native peers are filling the same vacuum — Cursor crossed $2 billion ARR in the same window — but the structural story is the repricing, not the ranking. Seat-based licensing, the model that underwrote two decades of developer software, is giving way to consumption metered against tokens, and value is migrating from the editor a human drives to the agent that does the work.

Why the margin moves to whoever owns the loop

The repricing has a unit-economic engine underneath it. An agent-assisted task costs roughly $6–7 of tokens against a fully-loaded knowledge-worker day of $350–500 — a 50–80x cost ratio before counting any quality or speed gain. And the cost side is deflating fast: Opus 4.5 hit the prior generation’s best SWE-bench score using about 76% fewer output tokens, so the cost to resolve a given issue is falling several-fold a year even as capability climbs. That is the inverse of the SaaS cost curve, where headcount and gross margin are roughly fixed. The ~75% gross margin seat-based software enjoyed was, in effect, a toll on human workflow; agents route around the workflow, and the surplus accrues to whoever owns the loop and the tokens beneath it.

Coding is the beachhead, not the destination

This generalizes beyond software because the loop does. Strip Claude Code to its pattern — read unstructured context, apply domain knowledge, produce structured output, verify against a standard — and you have the shape of most information work, on the order of a billion information workers and a $15-trillion-plus economy. Anthropic has already aimed the same architecture at general computing with Cowork, built by four engineers in ten days, most of its code written by Claude Code itself. The binding constraint is not model IQ; it is whether a task admits a cheap, trustworthy grader. Where one exists or can be built, expect the coding playbook to repeat; where it doesn’t, human judgment stays in the loop longer.

The bear case is specific, not vague. Verification debt: verifiable rewards optimize for passing the grader, not for what humans actually wanted, and as task horizons lengthen the graders get harder to write — reward hacking is reduced, not solved. Benchmark mirage: SWE-bench scores are compressing at the top (80.9% to 88.6% in months) and partly reflect contamination and scaffold tuning, so real-world generalization may trail the leaderboard. Capital intensity: the moat is expensive to hold — growth is now gated by compute and power, gross margin rides inference-cost and cloud-pricing curves Anthropic only partly controls, and the same RLVR recipe is available to every well-funded rival, so model leads are measured in months. The durable edge, if there is one, is not the model; it is the compounding system around it — proprietary verifiable environments, a hardened harness, enterprise distribution, and consumption pricing that turns usage into revenue.

Whoever writes the grader wins the category

The lesson of Claude Code is narrower and more useful than “AI is good at coding.” Software fell first because its grader was already written — the compiler and the test suite predate the model by decades. The verifiable-reward loop turned that free signal into capability, the harness turned capability into reliable work, and consumption pricing turned reliable work into the fastest revenue ramp software has seen. The next domains to fall will be the ones where someone writes the grader. The labs that build those environments — not merely the ones that ship bigger models — will capture the outcome.

Sources & notes

1. SemiAnalysis, “Claude Code is the Inflection Point” (Feb 2026) — 4% of public commits; ~135k commits/day; 20% projection; ACI and harness discussion. link

2. VentureBeat — Anthropic hits $30B run-rate after “crazy” 80x growth. link

3. Simon Willison — Anthropic run-rate revenue reaches ~$47B (May 2026). link

4. Anthropic — “Anthropic acquires Bun as Claude Code reaches $1B milestone” (fastest product in software history to $1B run-rate, ~6 months). link

5. Anthropic — Series G: $30B raised at $380B post-money valuation. link

6. CNBC — Anthropic tops OpenAI as most valuable AI startup (May 2026). link

7. MindStudio — Claude Code at ~$2.5B annualized (doubling in three months). link

8. Vellum / Abaka AI / Vals.ai — SWE-bench Verified: Opus 4.5 at 80.9% (first >80%), Opus 4.8 at 88.6%; competitor comparison. link · link

9. Anthropic — “Claude 3.7 Sonnet and Claude Code”: SWE-bench Verified 62.3% standard / 70.3% with scaffold. link

10. Anthropic engineering — SWE-bench agent: tool optimization, primitives over integrations, ~40% task-time reduction from tool-description rewrites; Model Context Protocol.

11. Related reading (author) — Zeoli, C., RL Environments for Agentic AI: Who Will Win the Training & Verification Layer by 2030, Data Gravity — source for the verification-as-bottleneck thesis and Anthropic’s RL-environment spend (tens of millions/yr, ~3–5x growth into 2026). link

12. METR — autonomous task horizons doubling every ~4–7 months (cited via SemiAnalysis).

13. getlatka / brandhistories — GitHub ~$2B run-rate; JetBrains ~$250M. link · link

14. GitLab Form 8-K (FY2024) — total revenue ~$579.9M. link

15. Software development tools market size (~$5–6B, 2024) — Verified Market Research; Cognitive Market Research. link

16. JetBrains April 2026 developer survey — Claude Code “most loved” (46%); at-work usage 3%→18%. link

17. VentureBeat — Anthropic: 80%+ of new production code authored by Claude; engineers merge ~8x more code/day; 76% success on hardest spec-light problems (+50 pts in six months). link

18. Anthropic — “When AI builds itself” (recursive self-improvement; share of merged code; 800+ fixes cutting one API-error class ~1,000x). link

19. Commit-share data (0.7% Sep ‘25 → 4.5% Mar ‘26; ~2.6M commits/week; 19M+ public commits; more than all other AI coding tools combined) — SemiAnalysis; serpsculpt; “Claude’s Code” tracker (Hacker News). link · link

20. Adoption / enterprise datapoints (Fortune 100, Accenture 30k, 54% coding LLM spend, enterprise >50% of Claude Code revenue, business seats ~4x YTD) — Menlo; Accenture newsroom; getpanto; TechnologyChecker. link · link

21. Fortune — Anthropic confidentially files for IPO at ~$965B after a $65B Series H (June 1, 2026); on pace for first profitable quarter. link

22. TechCrunch — Anthropic in talks to raise ~$50B at ~$900B (April 2026). link

23. FutureSearch — Anthropic financial forecast; Claude Code path toward ~$21B by May 2027 (input to the run-rate estimate in Fig. 6). link

24. Unit economics — agent-task (~$6–7) vs. loaded knowledge-worker day (~$350–500) and Opus 4.5 token efficiency (~76% fewer output tokens): SemiAnalysis; Anthropic. link

25. Figure 6 is the author’s estimate, not an Anthropic disclosure — triangulated from growth cadence, revenue-share, and the FutureSearch path. Treat the $6–12B band as the honest uncertainty.

The shift is measurable. On OpenRouter — the largest neutral router of LLM traffic, processing north of 20 trillion tokens per week — Chinese-origin models moved from a rounding error to the majority of consumption inside eighteen months. Anthropic and Google still command premium revenue per token, but on raw usage the center of gravity has moved offshore. This is what commoditization of a layer looks like in real time: the frontier becomes a feature, and the open, cheap, good-enough tier eats the volume.

This piece maps the field in three parts, plus the reframe the whole story turns on: (1) the labs and their relative share of use — Qwen, DeepSeek, Kimi, and the Alibaba/ByteDance platforms beneath them, including a deeper profile of who they are and what they are worth; (2) the chip ecosystem and supply — Huawei Ascend, the HBM bottleneck, and the offshore-neocloud workaround now being closed; and (3) ByteDance’s lead in video and media models, the one frontier where a Chinese lab is not catching up but setting the pace. Threaded through all three: the January 2025 DeepSeek panic predicted the wrong disruption.

Key points

• Distribution flipped. By May 2026, Chinese open-weight models were ~61% of all tokens consumed on OpenRouter, the largest neutral LLM router. Four of the five most-used models are Chinese — and Meta’s Llama, the open-weight leader two years ago, has fallen off the rankings entirely.

• The DeepSeek panic was backwards. Markets feared a cheap Chinese model would crater compute demand — they wiped ~$600B off Nvidia in a day. Nvidia is now worth $5.14T (+50% YoY). Semis didn’t break; the disruption landed on model market share instead.

• Price is the weapon. DeepSeek-V4-Pro lists ~12× under GPT-5.5 at comparable benchmark intelligence. Cost, not capability, is the wedge.

• The labs are a portfolio, not a monolith. Qwen owns the ecosystem, DeepSeek price-performance, Kimi agentic cadence, ByteDance media. The pure-play labs alone are now worth ~$160B combined; their platform backers add >$1.5T.

• Supply is HBM-bound, not logic-bound. SMIC can feed Huawei’s Ascend line; domestic HBM caps 2026 output near 250k–300k 910C-class packages. The binding constraint is memory.

• The offshore-compute loophole closed. On May 31, 2026, BIS extended licensing to any China-parented buyer regardless of where its subsidiary sits — shutting the Singapore/Malaysia neocloud channel for new purchases.

The share shift: China captured the open layer in eighteen months

The single most important chart in Chinese AI is not a benchmark — it is a usage curve. OpenRouter’s token-routing data shows Chinese open-weight models crossing from negligible to majority share of all tokens processed between late 2024 and mid-2026. The inflection traces directly to DeepSeek-R1 in January 2025, which proved that a Chinese lab could match Western reasoning quality and release the weights, and to the subsequent flood of Qwen, Kimi, GLM, MiniMax, and Xiaomi MiMo releases that filled every price-performance niche beneath the U.S. frontier.

Two structural forces sit under the curve. First, workload mix shifted toward code: programming rose from ~11% of OpenRouter usage at the start of 2025 to more than 50% by mid-2026, and Chinese models are disproportionately strong and cheap on coding. Second, the router itself scaled ~4× year-over-year — from roughly 5T tokens/week in April 2025 to over 20T in April 2026 — so China did not just take share, it took share of a market that quadrupled.

Aggregate share understates the drama at the top of the table. The incumbents did not merely lose ground proportionally; Google’s token share collapsed from roughly 37% to 13% over the year as developers re-routed cost-sensitive workloads to open weights. Anthropic held the premium tier on reasoning and coding quality, and OpenAI’s share of routed volume — distinct from its first-party ChatGPT traffic — fell into single digits.

Read the vendor table closely and three things stand out. First, the single largest vendor is Xiaomi — not a name most Western investors associate with frontier AI — whose MiMo models alone take ~21% of routed tokens and ~22% of all coding traffic. Second, DeepSeek at ~17.6% outranks every U.S. lab, sitting above Anthropic’s 15.4% and well above Google and OpenAI. Third, Meta has effectively exited: the company that defined open-weight AI with Llama in 2023–24 has fallen below 1% of routed volume, displaced entirely by Chinese open weights. (Methodologies differ — counting only identified vendors puts the Chinese share near 56%, while the broader origin-based measure in Exhibit 1 reaches ~61% — but every cut tells the same story.)

The frontier is no longer where the volume is. Premium reasoning is a 15-point niche; the cheap, open, good-enough tier is the market — and it is overwhelmingly Chinese.

One caveat worth stating plainly: routed inference is not the whole market. First-party consumer traffic — ChatGPT, Gemini, Doubao — dwarfs the router, and Anthropic and OpenAI still capture the majority of industry revenue because they price at a premium and serve enterprises directly. OpenRouter is a leading indicator of where developers reach when nobody is watching the brand. On that signal, the open Chinese tier is the default.

The DeepSeek panic predicted the wrong disruption

It is worth remembering what the market actually believed eighteen months ago. On January 27, 2025, days after DeepSeek-R1 showed a Chinese lab could match Western reasoning for a fraction of the training cost, investors concluded the obvious: if frontier intelligence is suddenly cheap, the demand for expensive accelerators must collapse. They sold. Nvidia fell 17% and shed roughly $600 billion in market value in a single session — the largest one-day loss in U.S. market history — and Broadcom dropped 17% (~$200B). The thesis was crisp: efficient Chinese models break the compute supercycle.

That thesis was wrong in the most complete way possible. By June 2026, Nvidia is worth $5.14 trillion, up ~50% year-over-year, the most valuable company on earth, and AI capex hit records on both sides of the Pacific. Cheaper intelligence did not shrink compute demand — it expanded it. Token volume on OpenRouter alone quadrupled; total inference grew faster than per-token efficiency improved. The Jevons dynamic held: drop the unit cost of intelligence and the world consumes vastly more of it.

What actually came true was the other half of the DeepSeek thesis — the half the market ignored while it was selling chips. DeepSeek did not destroy demand for compute. It demonstrated that Chinese labs would capture the model layer, and they did: from ~2% to ~61% of routed tokens. The disruption was real, but it landed one layer up from where the market priced it — on model-vendor share, where margins were already thin and are now mostly Chinese, not on silicon, which compounded. For an infrastructure investor, that is the single most important re-rating of the cycle: efficiency gains accrue to the compute layer and the distribution layer; they get competed away in the middle.

The Chinese Labs: A portfolio of specialists, not a single champion

Treating “Chinese AI” as one actor obscures the real structure. It is a portfolio of at least seven serious franchises, each with a different control point, business model, and owner. Three pure-play open-weight labs — DeepSeek, Moonshot (Kimi), and Zhipu (GLM) — compete on weights, price, and release cadence. Two platform giants — Alibaba (Qwen) and ByteDance (Doubao/Seed) — treat models as demand generation for cloud and consumer surfaces. And two more — MiniMax and Xiaomi (MiMo) — have become major suppliers almost sideways, the first out of a content-generation business, the second out of a phone company. What they share is a strategy: release open weights, price at a fraction of the U.S. frontier, and let distribution compound.

The aggregate is the statistic that should reframe the category for any investor still treating these as research curiosities. The four pure-play Chinese AI labs — Zhipu, DeepSeek, MiniMax, and Moonshot — are now worth roughly $159B combined, up from a near-zero base eighteen months ago. DeepSeek alone went from a quant fund’s side project to a reported ~$50B first external round (founder Liang Wenfeng personally subscribing ~$3B, ~40% of it). Behind them stand platform owners — Tencent, ByteDance, Alibaba — carrying more than $1.5 trillion of AI-exposed market value.

Qwen / Alibaba — the ecosystem standard

Qwen is the most important open-model franchise in the world, and the gap is not close. It has surpassed 1 billion cumulative downloads on Hugging Face, overtaking Meta’s Llama as the most-downloaded open model, and now anchors more than 200,000 Qwen-tagged models and over 113,000 derivatives — more than Google and Meta’s base families combined. Roughly 40% of all new LLM derivatives created on Hugging Face are now Qwen-based, and China accounted for ~41% of all Hub downloads over the trailing year. In June 2026 trending rankings, Chinese open models held five of the top ten slots.

The strategic logic is not philanthropy. Alibaba runs Qwen as a top-of-funnel for Alibaba Cloud: open weights seed adoption, paid inference and fine-tuning capture the monetizable tail. It is working. Cloud Intelligence Group revenue reached ¥158.1B in FY2026 (+34% YoY), AI-related product revenue posted triple-digit growth for eleven consecutive quarters, AI now drives ~30% of external cloud revenue (a ~$5B run-rate), and the firm pushed capex to ¥126.1B for the year. Open weights are the cheapest customer-acquisition channel in infrastructure.

DeepSeek — the price-performance frontier

DeepSeek is the lab that started the repricing. Founded in July 2023 by Liang Wenfeng as an offshoot of his quantitative hedge fund High-Flyer, it has no consumer platform to defend and no cloud P&L to protect — just a model franchise, a stockpile of GPUs the fund had accumulated for trading, and a founder who treats open weights as ideology. Its R1 release in January 2025, trained for a reported <$6M on export-compliant H800s, is what triggered the Nvidia selloff. In 2026 it took its first outside capital at a reported ~$45–50B valuation, with Liang personally subscribing ~$3B — a structure that keeps control firmly with the founder.

The product strategy is to undercut. The V4 family (April 2026) spans V4-Pro (1.6T parameters, 49B activated) and V4-Flash (284B / 13B activated), both at 1M-token context. List pricing of $0.435 / $0.87 per 1M input/output tokens places it roughly 3× below Gemini 3.1 Pro and ~12× below GPT-5.5 at comparable benchmark intelligence. The result is measurable U.S.

pull: DeepSeek’s share of token usage on Vercel rose from under 1% to ~17% by May 2026, even as its share of revenue stayed near 1% — the clearest single illustration of how cheap the open tier has made intelligence.

Kimi / Moonshot — agentic coding and cadence

Moonshot, founded in March 2023 by Tsinghua- and Carnegie Mellon-trained researcher Yang Zhilin, competes on two axes incumbents struggle to match: coding quality and shipping speed. Kimi K2.6 (April 20, 2026), a 1T-parameter MoE, ties GPT-5.5 on SWE-Bench Pro (58.6%) and leads Humanity’s Last Exam with tools (54.0) — ahead of GPT-5.4, Claude Opus 4.6, and Gemini 3.1 Pro on that benchmark — at roughly 80% lower cost, with an agent-swarm architecture that coordinates up to 100 parallel sub-agents. Artificial Analysis ranks it #1 among open-weight systems. Then came K2.7-Code on June 12 — the fifth major Kimi release in under a year, cutting reasoning-token usage ~30%. That cadence is the moat: Moonshot iterates faster than enterprise procurement cycles.

The market is paying for it. Moonshot raised $2B at a $20B valuation in May 2026 (cumulative ~$3.9B in six months, the most of any Chinese LLM startup), with Alibaba and Tencent on the cap table, as revenue ran from a $100M annualized run-rate in March to $200M in April. Of the pure-play labs, Moonshot has best converted benchmark leadership into commercial momentum.

ByteDance — scale, media, and the cloud beneath it

ByteDance is the platform counterweight to the open-weight labs, and — at a ~$480B secondary-market valuation (General Atlantic has tested $550B) — by far the most valuable of them. Its Doubao assistant reached 345M monthly active users and 120 trillion daily token calls, and its Volcano Engine cloud lifted its model-as-a-service revenue target to ¥15B (~$2.2B). ByteDance keeps its best models proprietary and behind APIs — the opposite of Qwen — because it monetizes attention and cloud, not weights, and it is far enough along on a self-designed inference chip (targeting 100k+ units in 2026) to hedge its Nvidia dependence. Its decisive lead, covered in Section 4, is in video.

The next tier — GLM, MiniMax, and the Xiaomi surprise

Three more names matter for anyone sizing the supply base. Zhipu (Z.ai), a 2019 Tsinghua spin-off now listed in Hong Kong at ~$56B, shipped GLM-5.2 — a 744B-parameter model that leads the Artificial Analysis Intelligence Index among open weights in June 2026, and is the strongest of the cohort in enterprise and government deployments. MiniMax (founded 2021 by ex-SenseTime researcher Yan Junjie; HK-listed ~$33B) pairs the M-series LLMs with its Hailuo video/audio models, straddling text and media. And the genuine surprise is Xiaomi: its MiMo models, an afterthought to a phone-and-EV company, are by several weekly counts the single most-used vendor on OpenRouter (~21% of tokens, ~22% of coding traffic). That a handset maker can lead global open-model usage is the strongest evidence that distribution and price, not research prestige, now decide share.

Chips and Supply: Huawei silicon, an HBM ceiling, and the offshore workaround

The software story above runs on hardware these labs increasingly cannot buy at will. China’s compute supply now rests on three legs: a maturing domestic accelerator in Huawei’s Ascend line, a still-open but narrowing channel to Nvidia silicon, and — until five weeks ago — a large offshore rental market in Southeast Asia. Each leg has a different fragility.

Huawei Ascend: competitive at the rack, constrained at the wafer’s edge

Huawei’s CloudMatrix 384 (CM384) is the clearest evidence that China can build at the system level. It lashes 384 Ascend 910C dies into a single super-node delivering roughly 300 PFLOPs of dense BF16 — close to 2× the compute of Nvidia’s GB200 NVL72 — by trading power efficiency for brute scale. At the rack, China has an answer. The constraint sits one layer down. Per SemiAnalysis, SMIC’s logic capacity is no longer the binding limit; high-bandwidth memory is. Domestic HBM from CXMT is projected at ~2.0M stacks in 2026, enough to finish only ~250,000–300,000 910C-class packages.

For investors, the takeaway is that China’s accelerator output is small relative to demand — a few hundred thousand high-end packages against domestic compute appetite measured in millions of GPU-equivalents — and that the leverage point for U.S. policy has moved from logic (SMIC, lithography) to memory (HBM, CXMT/Samsung/SK Hynix). The 910C is also less power-efficient than Nvidia’s parts, so China is spending its scarce energy and capital budget less efficiently per useful FLOP. That gap is closing, but it is real and it is the reason the offshore-Nvidia channel mattered so much.

The capex arms race

Constrained supply has not slowed spending — it has accelerated it, as the platform owners race to lock up whatever compute they can. Alibaba lifted full-year capex to ¥126.1B (≈ $17.5B), up from ¥84.3B. ByteDance is the more aggressive spender, raising 2026 capex past ¥200B (≈ $28–29B), with roughly half historically earmarked for AI chips — and some reports put its ceiling materially higher.

The offshore-compute loophole — opened, exploited, and just closed

Because the best Nvidia parts were export-restricted and Huawei output was capped, Chinese labs did the rational thing: they rented frontier GPUs offshore. For roughly a year, a China-parented company could stand up — or rent from — a data center in Singapore, Malaysia, or Indonesia and run banned Blackwell-class silicon legally, because the hardware never shipped to the mainland. ByteDance reportedly secured access to a 36,000-GPU Blackwell cluster through a Malaysian cloud operator, a deal Nvidia confirmed as compliant. That channel narrowed sharply on May 31, 2026, when BIS extended licensing to any buyer whose ultimate parent sits in China or Macau, regardless of where the subsidiary is registered.

The net picture on compute: China can build competitive systems but cannot yet make enough of the memory that finishes them, the cheapest path to frontier silicon (offshore rental) just narrowed, and the policy lever has migrated to HBM. Expect three responses — a domestic HBM build-out (CXMT capacity expansion), continued Ascend system scaling despite efficiency penalties, and more inference pushed onto the very open models from Section 2, which are cheap enough to run on constrained hardware. Scarcity of training compute is precisely why the open, efficient model tier is a strategic asset, not just a commercial one.

ByteDance and the video frontier: Not catch up — it is ahead

In text and code, the Chinese labs win on price and distribution while the U.S. frontier still sets the quality ceiling. In video and media generation, that ceiling is now Chinese. The shift is partly capability and partly economics: video models are absurdly expensive to serve, and the firms that win are the ones that can subsidize generation against another business — ads, short-form, a billion-user app. ByteDance has exactly that, and it has used it to take the lead.

ByteDance’s Seedance 2.0 (February 12, 2026) is a unified multimodal audio-video model: text, image, audio, and video in; synchronized video plus dual-channel stereo audio out, in a single forward pass, with native multi-shot inside one 15-second render. It topped Artificial Analysis’s Video Arena for both text-to-video and image-to-video from launch, and remains #1 on the with-audio board. Its only real challenger is also Chinese — Alibaba’s HappyHorse-1.0 — with Google’s Veo 3.1 holding third and Kuaishou’s Kling 3.0 placing four entries in the top ten.

Why the economics favor ByteDance, not OpenAI

The clearest proof that media generation is an economics problem, not just a quality problem, is Sora’s collapse. OpenAI’s consumer video product reportedly burned ~$15M per day in compute against ~$2.1M in total lifetime revenue — a day of cost exceeding all the revenue it ever earned by roughly 7×. OpenAI deprecated Sora in April 2026. ByteDance runs the same workload at the same scale, but against a short-video advertising machine that already monetizes attention. Seedance is not a standalone P&L; it is a feature of Doubao, Jimeng (Dreamina), Volcano Engine Ark, and BytePlus.

That structural subsidy is why ByteDance can keep a video model at the frontier while a pure-play cannot. It also explains ByteDance’s monetization posture: keep the models proprietary, expose them through APIs and apps, and capture value at the consumer and cloud layers. Volcano Engine raised its model-as-a-service revenue target to ¥15B on the strength of Seedance demand, and Doubao’s reach gives every model instant distribution.

Implications: Where margin survives when the model layer is free

The strategic conclusion is not “China is winning AI.” It is narrower and more useful: the open model layer has been commoditized, and China supplies it. That repricing is permanent, and it relocates where value accrues. For an infrastructure investor, five implications follow.

1. Weights are a funnel, not a moat. Qwen’s billion downloads are a customer-acquisition expense for Alibaba Cloud, not a product. Any thesis that underwrites a standalone open-weight model business as the durable asset is mispriced. The defensible positions are distribution (ByteDance, Alibaba), cost structure (DeepSeek), release cadence (Moonshot), and the cloud/inference layer that monetizes all of them. Margin lives above and below the model, not in it.

2. The binding constraint is memory, and so is the policy lever. China can design competitive systems (CM384) and has adequate logic capacity (SMIC), but domestic HBM caps 2026 high-end output near 250k–300k packages. That makes HBM the single highest-leverage node in the entire stack — for Beijing’s industrial policy, for Washington’s controls, and for investors. Watch CXMT’s ramp, Samsung/SK Hynix’s China exposure, and any HBM-specific export action; those move the real frontier more than any model release.

3. Scarcity of training compute is exactly why efficient open models are strategic. When you cannot buy unlimited frontier silicon, a model that delivers near-frontier quality at a tenth of the inference cost is not just commercially attractive — it is a national hedge. The cheapness that looks like a race to the bottom is, from Beijing’s side, a way to extract maximum useful intelligence from constrained hardware. Expect more inference pushed onto small, efficient, open models and more domestic Ascend deployment despite its efficiency penalty.

4. In media, back the distribution, not the generator. Sora’s shutdown is the cleanest data point of the year: a frontier-quality video model with no attached business is uninvestable at current unit economics. The winners are models welded to a consumer or ad platform that can subsidize generation — ByteDance first, Kuaishou and Alibaba behind it. Standalone video pure-plays are structurally short of the cash flow needed to stay at the frontier; the durable plays are the platforms and the inference infrastructure underneath them.

5. The DeepSeek panic is the template: price the disruption to the right layer. The market’s January 2025 error was not believing China would disrupt — it was putting the disruption on silicon when it belonged on model-vendor share. Efficiency gains flowed down to compute (Nvidia at $5.14T) and up to distribution; they were competed away in the middle, where the open Chinese labs now sit. That is also the risk to the labs themselves: at a combined ~$160B, the pure-plays occupy the thinnest-margin layer in the stack, and their equity value depends entirely on converting free weights into something that isn’t free — Alibaba’s cloud pull, ByteDance’s consumer surface, Moonshot’s enterprise ARR. For a U.S. investor, the cleanest hedge against Chinese model-share gains is to own the compute and memory layers that every one of those tokens still runs on.

What to watch over the next 12–18 months

• HBM: CXMT capacity beyond ~2M stacks; any HBM-specific U.S. control. The output ceiling moves with this number.

• Offshore enforcement: whether the May 31 BIS parent-company rule is actually enforced against Singapore/Malaysia/Gulf neoclouds, or merely announced.

• Frontier gap: whether DeepSeek/Kimi close the remaining quality gap to GPT-5.5 / Claude Opus on agentic tasks, or plateau just below it.

• Monetization: Alibaba Cloud AI run-rate (past ~$5B) and ByteDance Volcano MaaS revenue (¥15B target) — the proof that open weights convert to cloud margin.

• U.S. response: whether a credibly open, efficient U.S. model re-enters the Hugging Face top tier, or the open layer stays Chinese by default.

Summary

As of June 2026, China supplies the majority of the world’s open-model tokens, sets the price of intelligence at the open tier, and leads outright in video generation. It does this from a constrained compute base — competitive systems, capped memory, a just-closed offshore channel — which is precisely why efficiency and openness are central to its strategy rather than incidental to it. The U.S. frontier still defines the quality ceiling and captures most industry revenue, but it no longer controls the default. The contest has moved from “who builds the best model” to “who controls the layers that still carry margin”: memory and interconnect below the model, distribution and cloud above it. That is where the next five years of returns — and the next round of export policy — will be decided.

Sources

1. OpenRouter — State of AI: 100T-Token LLM Usage Study and LLM Rankings.

2. KuCoin / Global Times — OpenRouter: 61% of token consumption by Chinese models; Chinese models take top six usage spots.

3. digitalapplied — OpenRouter Rankings April 2026; Chinese AI Models Q2 2026 Landscape.

4. Hugging Face — State of Open Source, Spring 2026; One Year Since the DeepSeek Moment.

5. Pandaily — Qwen surpasses 1 billion downloads; getaibook — Chinese open models overtake U.S. on the Hub.

6. DeepSeek — API Pricing; verdent.ai — V4 Pricing & Migration (2026); sitepoint — DeepSeek V4 Released.

7. Moonshot AI — Kimi K2.6; miraflow — K2.6 ties GPT-5.5 on coding; DevOps.com — Kimi K2.7-Code.

8. SemiAnalysis — Huawei Ascend Production Ramp (HBM bottleneck); CloudMatrix 384; Tom’s Hardware — Ascend ecosystem scales.

9. Alibaba — FY2026 6-K; CIW — Alibaba Cloud Q1 2026. ByteDance capex — EqualOcean, SCMP.

10. Export controls / neoclouds — Tom’s Hardware — ByteDance 36k Blackwell via Malaysia; Asia Times — SE Asia data-center crackdown; Indoneo — BIS parent-company rule.

11. ByteDance video / Doubao — ByteDance Seed models; sitepoint — Seedance 2.0; kr-asia (Volcano MaaS target); Sacra — ByteDance overview. Video market — digitalapplied: AI video after Sora; Bloomberg.

12. DeepSeek panic / Nvidia — CNBC — Nvidia sheds ~$600B, biggest one-day loss; IG — why NVDA fell 17%; MacroTrends — NVDA market cap ($5.14T, Jun 2026).

13. Vendor share / Meta decline — stockalarm — OpenRouter rankings investor analysis (Chinese 46–56%); cryptobriefing — U.S. startups shifting to Chinese LLMs.

14. Valuations — TechCrunch — Moonshot $2B at $20B; SCMP — DeepSeek first raise; Bloomberg — ByteDance ~$480B; companiesmarketcap — Chinese company market caps.

The dominant narrative of the last software cycle was that value migrates up the stack — toward the application, the workflow, the system of record, the seat you sell. That was true for SaaS, where the marginal cost of serving one more user approached zero and durable 75–90% gross margins accrued to whoever owned the interface.

AI inverts it. The center of gravity is moving down the stack, toward the layers with the highest capital intensity, the deepest physical constraints, and the strongest control points: semiconductors and compute, data platforms, open models and the inference engines that run them, and a narrow premium frontier tier. The application layer does not disappear — but it thins. Its margins compress as inference becomes a variable cost of goods, and its multiples bifurcate: the model and the data re-rate upward, the wrapper does not.

Stated plainly: the future is a very large number of open-source models (with inference platforms to run and optimize them), a smaller set of premium models, data platforms, and the semiconductors to power all of it — sitting beneath a thinner application software layer than the SaaS era trained us to expect.

The rotation reverses

This is heretical to how private capital has thought for twenty years. After the dotcom crash, investors fled capital-intensive, cyclical semiconductors and rotated decisively into asset-light cloud and SaaS — 80%+ gross margins, near-zero marginal cost, recurring revenue, and capital efficiency that compounded into the highest multiples in the market. “Software is eating the world” became the organizing thesis of an entire generation of funds; the LP and GP base reorganized around application software, and the semiconductor industry was treated as a low-multiple, cyclical afterthought.

AI reverses the rotation. Capital is flowing back down into the physical, capital-intensive layers the industry spent two decades fleeing — silicon, packaging, power, data infrastructure, and the inference plumbing on top. The most valuable franchise in technology is, again, a chip company; the fastest-growing infrastructure businesses ever recorded are inference and data platforms. The asset-light application layer — the destination of the last rotation — now carries the worst margin structure in the stack. Twenty years of muscle memory points the wrong way.

Framework: value has gravity

Treat the AI stack as five layers and ask one question of each: how easily can the next dollar of value be substituted away? Value, like mass, accumulates where it cannot easily move. Four properties pull value down the stack and hold it there.

1. Capital intensity. The base layers cost tens of billions to build, continuously. The four largest hyperscalers are spending at a $500B+ annualized pace on capex, heading toward $600B+ for 2026. Capital is a moat when it is both enormous and perpetually required.

2. Physical bottlenecks. Advanced packaging (CoWoS), high-bandwidth memory, and power are hard limits that money cannot relax on a quarterly cadence. Scarcity accrues to whoever controls the constraint.

3. Switching cost and inertia. Data has gravity in the original sense: every pipeline, table, and model trained in place raises the cost of leaving. Open weights are the opposite — engineered to be portable — which is exactly why value at that layer shifts from the model to the platform that serves it.

4. Replacement risk at the surface. Generic application logic is the most substitutable thing in the stack, because the model can increasingly perform it directly.

The rest of this piece walks the layers from the base up, with the numbers.

Layer 1 — Semiconductors and compute: the densest mass

This is the heaviest layer, and it is getting heavier. In its most recent quarter — ended April 2026 — NVIDIA’s data-center revenue hit $75.2B, up 92% year over year, at a ~75% gross margin: roughly a $300B annualized run-rate, with the current quarter guided to $91B in total revenue. NVIDIA holds about 80% of AI accelerators by revenue. Those are not the economics of a competitive commodity market; they are the economics of a control point.

The demand beneath it is a capex supercycle without parallel in enterprise technology. In Q1 2026 alone, the four largest hyperscalers spent roughly $131B on capex — Amazon $44.2B, Alphabet $35.7B, Microsoft $30.9B, Meta $19.8B — a ~$525B annualized pace, still climbing toward $600B+ of full-year 2026 guidance. Goldman Sachs models $5.3T of cumulative hyperscaler capex from 2025 through 2030.

And it is not only compute. The entire semiconductor industry is in a supercycle: global revenue hit $793B in 2025 (+21% YoY) and is on track to approach $1 trillion in 2026 (WSTS ~$975B, +25%; Gartner higher still), with AI now ~30% of all semiconductor revenue and headed past 50% by 2029. The demand splits into three buckets, and all three are supply-constrained. Compute — logic, GPUs, and accelerators — is the largest: AI accelerators alone scale from ~$80B in 2024 to $280B+ by 2029. Memory is the fastest-moving: HBM demand grew ~130% in 2025 and another ~70% in 2026, conventional DRAM contract prices jumped ~90% quarter-over-quarter in Q1 2026, and AI data centers are absorbing roughly 70% of high-end DRAM. Storage is quietly tightening too: the NAND market reaches ~$65B in 2026, one in five bits now feeds AI, and demand (+20–22%) is outrunning supply (+15–17%) into 2027. When a single end-market reprices DRAM 90% in a quarter and books advanced packaging two years out, the constraint — and the margin — has moved into the physical layer.

Two features make this layer durable rather than merely large. First, the binding constraint has moved from chip design to packaging and power. TSMC’s CoWoS capacity — about 70k wafers/month in 2025, rising toward ~110k in 2026 — is effectively sold out, and NVIDIA has reserved the majority of supply into 2027. Second, more than 10 GW of new AI data-center capacity was announced in 2025, against a packaging base that can support on the order of 18 GW. When the bottleneck is physical, capital intensity is a feature for incumbents and a wall for entrants.

The replacement risk here is real but slow: hyperscaler custom silicon (Google TPU, AWS Trainium, Broadcom ASICs) is growing into a $200B+ accelerator TAM. It compresses NVIDIA’s share over time; it does not move value out of the layer. The mass stays at the bottom.

The real risk at this layer is returns, not revenue. At a $500B-plus annual capex run-rate against accelerators that depreciate over few year useful lives, margin durability depends on utilization staying high enough to earn a return on the installed base. That — not competition — is the single largest swing factor beneath the entire stack.

Layer 2 — Open models and inference: abundance at the model, value at the platform

The single most important quantitative story in AI over the past 24 months is convergence. Per the Stanford AI Index, the gap between the best closed model and the best open-weight model on Chatbot Arena fell from 8.04% in January 2024 to 1.70% in February 2025. On MMLU, the 2023 gap of ~17.5 points has effectively closed. DeepSeek R1 showed that frontier-class reasoning does not require nine-figure training budgets; Qwen overtook Llama as the most-downloaded model family on Hugging Face.

The supply side is now effectively infinite. Hugging Face hosts more than 2.2M models, with the catalog doubling in roughly 24 months. When a capability exists in thousands of near-equivalent open variants, the weights themselves stop being the scarce asset. The scarce asset becomes the ability to run them well — throughput, latency, cost per token, routing, caching, fine-tuning, evaluation, and guardrails.

There is a market-structure wrinkle investors should not miss: the open-weight frontier is increasingly non-US. Chinese developers accounted for 17.1% of Hugging Face downloads from August 2024 to August 2025, edging out the US at 15.8%, and Qwen displaced Llama as the most-downloaded family. The commoditizing layer is also the layer least controlled by US incumbents — a supply-chain and policy variable, not only an economic one.

Linux did not capture the value of Linux; Red Hat and the cloud did. The same pattern now applies to open weights — value accrues to the platform that serves them, not the model itself.

This is why the durable business at this layer is the inference and optimization platform, not the open model. The platform sits between abundant open supply and enterprise demand, and monetizes the one thing that does not commoditize: making a portable artifact perform reliably and cheaply in production.

The market is proving this in real time, at a velocity with no precedent in infrastructure software. Fireworks AI disclosed it crossed $800M in annualized revenue in roughly three years — up over 4× year over year. Together AI is rumored to be at a similar scale; Baseten reached ~$600M, up >5X year over year. None of these are model labs — they are the toll roads that make open weights usable in production. When the model itself is free, the company that serves it fastest and cheapest captures the spend.

The math is why. Consider a team serving 50B output tokens a month — a mid-size AI feature. At frontier-API output pricing of ~$10–15 per million tokens, that runs ~$500–750K a month; the same workload on an open model (Llama, Qwen, DeepSeek) served through an inference platform at ~$0.40–1.00 per million costs ~$20–50K — a 90%+ reduction. Below a quality threshold that keeps falling, open-plus-inference wins on cost by an order of magnitude, and the platform that delivers it keeps a slice of the savings as margin. That is the whole business. (Illustrative; actual costs vary with model, context length, and utilization.)

Layer 3 — Premium models: a smaller, real, defensible tier

Convergence compresses the average but not the frontier. A premium tier persists wherever the marginal point of capability — complex agentic tasks, long-horizon reliability, production coding, safety — is worth paying for. Revenue proves it: Anthropic reported ~$47B annualized by May 2026 (from ~$1B at end-2024) and is tracking toward an estimated ~$100B run-rate by year-end; OpenAI — ~$6B at end-2024, ~$20B at end-2025, ~$30B in mid-2026 — is on a path to an estimated ~$60B. More than 1,000 enterprises spend over $1M/year with Anthropic alone.

(Both labs report on different revenue bases, and year-end 2026 figures are estimates; treat the absolute numbers as directional and the trajectory as not.)

The strategic point for investors: this is a narrower layer than the 2023 consensus assumed. Open weights have taken the floor; the premium tier monetizes the ceiling. The defensible position is the frontier plus the distribution to reach enterprises — not “a good model,” which is now free and plentiful.

Layer 4 — Data platforms: the gravity well that compounds

If semiconductors are the densest layer, data platforms are the stickiest. This is “data gravity” in its literal form: the cost of moving away rises with every pipeline, table, and ontology built in place.

The numbers show the layer capturing the AI workload directly — and the market paying premium multiples for it. Databricks crossed a $6.9B annualized run-rate growing ~80%, with AI products already ~26% of revenue, valued around $170B privately. Palantir — the operational layer that turns enterprise data into governed decisions — grew revenue 85% year over year in Q1 2026 (its fastest ever) to a ~$6.5B run-rate, with US commercial revenue up 133% and a Rule of 40 of 145%. At a ~$350B market cap, the market pays Palantir a foundation-model-class multiple — roughly 50× revenue — for owning the decision loop that sits on top of data.

Why does value hold here while it leaks at the model layer? Because data is the one input that is both non-substitutable and accretive. Models converge; a company’s proprietary data — and the workflows and ontology built on it — does not. As the model layer commoditizes, the platform that owns the data and the decision loop becomes the durable point of leverage. Note the inversion this implies for the application thesis: the layers commanding the highest multiples — Palantir at ~50× revenue, foundation models at 25–50× — are precisely the ones fused to proprietary data, not the generic software surface.

Layer 5 — The application layer: thinner than the SaaS playbook assumes

Now the top. The thesis is not that applications die; it is that the generic application surface thins, in two measurable ways.

First, margins. Traditional SaaS runs 75–90% gross margins because marginal serving cost approaches zero. AI-native applications run 50–60%, because every query re-runs a model. By ICONIQ’s 2026 data, inference averages ~23% of revenue at scaling AI companies — and, critically, it does not fall with scale the way SaaS COGS did. The thin-wrapper extreme can sit at 25% gross margin. Inference is a structural, variable cost of goods.

Second, multiples. The market is already pricing the bifurcation. Foundation-model companies trade at 25–50× revenue; AI-native platforms at 25–30×; traditional SaaS has compressed to a ~6.7× median (from an 18.6× peak in 2021); and “AI wrappers” sit at 5–8× — at or below legacy SaaS. Value re-rates to the model and the data; it derates at the undifferentiated surface.

This does not mean “don’t invest in applications.” It means the bar moved. A durable application now needs something the model cannot absorb: a proprietary data loop, a system of record, a regulated workflow, real distribution, or outcome-based pricing that converts inference cost into margin. Absent that, the application is a thin layer of prompt-and-glue over a commoditizing model — and the multiple will say so.

The counter-thesis: what would make me wrong

Decisive does not mean unfalsifiable. Three developments could redistribute value back toward the top.

Apps capture the data loop. If the best applications accumulate proprietary interaction data fast enough, they become data platforms in their own right and inherit the gravity. This is the strongest bull case for the application layer, and the reason the system-of-record app is exempt from the thinning.

Vertical integration collapses layers. A premium-model lab that owns inference, builds agents, and sells the application directly could re-internalize value across three layers at once. That concentrates value rather than redistributing it upward — but it does erode the independent inference and app layers.

Inference commoditizes faster than expected. LLMflation — roughly a 10×/year decline in the cost of a unit of intelligence, with median price drops near 200×/year since early 2024 — cuts both ways. If serving becomes nearly free, AI-app gross margins recover toward SaaS norms and the margin half of the thinning thesis weakens. (The multiple half largely holds, because substitutability, not cost, drives it.)

Note the tension: the same cost collapse that pressures the semiconductor and inference layers on price is what could make the application layer’s margin problem temporary. I weight the structural argument above the cost-relief argument, because substitution risk at the surface persists even at zero inference cost.

Implications

For investors. Underwrite control points, not interfaces. The highest-conviction, lowest-substitution positions are: compute and the physical bottlenecks around it (packaging, power, memory); data platforms with proprietary accretion; and the inference/optimization layer that monetizes open-model abundance. In the premium-model tier, underwrite the frontier plus distribution, and size for a narrower-than-consensus winner set. At the application layer, require a data loop or a system of record before paying a software multiple — and assume 50–60% gross margins, not 80%+, until proven otherwise.

For founders. If you are building an application, your job is to manufacture gravity. Own a proprietary data loop, become a system of record, or price on outcomes so that falling inference cost flows to your margin rather than your customer’s savings. “Thin” describes the default, not a sentence.

For incumbents. The capital intensity of the base layers is a moat you can rent but not own. For most enterprises the defensible move is up at the data layer — keeping data, training, and serving in one place — rather than competing on compute or frontier models.

What would confirm — or break — this

Five measurable signals over the next four quarters will tell you whether the heavy stack is holding:

• Hyperscaler capex-to-revenue. If it keeps climbing past ~25–30% of revenue without denting margins, the base layer’s returns thesis holds; if ROIC cracks, the whole stack reprices.

• Inference-platform ARR. Fireworks, Together, and Baseten compounding toward ~$1–2B each confirms value is pooling at the inference layer, not the model.

• The open–closed gap. If it stays inside ~2 points on hard benchmarks (agentic, coding), commoditization of the model layer is locked in.

• AI-app gross margins. If scaling AI apps stay at 50–60% (inference not falling with scale), the thinning thesis holds; a recovery toward 75%+ would weaken it.

• DRAM and HBM pricing. Continued double-digit quarterly price increases mean the physical constraint — and the margin — is still at the base.

Summary: probability-weighted, 3–5 years

The recommendation is consistent across the high-probability branches: position at the base and the data layer, treat the premium-model tier as a narrow high-conviction bet, and hold applications to a higher bar — a data loop or a system of record — before underwriting them at software multiples. Gravity is doing the work. Invest where the mass is.

Sources: NVIDIA Q1 FY2027 results (May 2026); company Q1 2026 earnings (Amazon, Alphabet, Microsoft, Meta); Stanford HAI AI Index 2025; Hugging Face; Epoch AI; Andreessen Horowitz ("LLMflation"); Gartner and WSTS (semiconductor revenue); TrendForce and Counterpoint (DRAM/NAND/HBM); BofA; Databricks and Palantir disclosures; OpenAI, Anthropic, Fireworks AI, Together AI, Baseten (via Sacra / PYMNTS); ICONIQ State of AI 2026; Goldman Sachs; ValueAddVC; SaaS Capital. Private-company revenue run-rates are reported on differing bases; year-end 2026 model-lab figures are estimates.

Compute Express Link (CXL) is an open, cache-coherent interconnect that lets CPUs, accelerators, and memory devices share one address space over the PCIe physical layer. It exists because memory stopped scaling at the rate compute does, and because the unit of AI infrastructure spend has shifted from FLOPS to bytes held in fast memory.

CXL is the mechanism for decoupling memory capacity from the CPU socket it used to be soldered next to — turning memory into a pooled, fabric-attached resource instead of a fixed, stranded one. Seven years after its 2019 launch, it has gone from a side project to the default answer for memory disaggregation: all three big DRAM makers build CXL memory modules, every major CPU exposes CXL root ports, and the spec hit its fourth major revision in November 2025. The timing is not incidental. With DRAM prices up roughly 90% quarter-over-quarter in Q1 2026 and HBM crowding conventional DRAM off the wafer, the economic case for reclaiming stranded memory just went from theoretical to urgent.

Key Points

What is CXL in simple terms?

Compute Express Link is an open standard that lets a CPU treat memory attached over a PCIe cable or slot almost like its own local RAM — readable with ordinary load/store instructions, kept cache-coherent — so capacity and bandwidth can be added, pooled, and shared instead of being fixed to each socket.

What problem does CXL solve?

The memory wall. Compute has scaled ~600× faster than memory bandwidth over two decades, and AI inference (especially KV cache) consumes memory per user. CXL adds a cheaper, poolable memory tier and recovers DRAM that would otherwise sit stranded.

Is CXL used to connect GPUs to each other?

No. GPU-to-GPU scale-up is the domain of NVLink and UALink. CXL targets CPU-to-memory expansion, pooling, and coherency — the two are complementary, not competing.

Who are the main CXL companies?

Astera Labs (Leo controllers), Marvell (Structera controllers and switch), Credo (SerDes/retimers), Micron, Samsung and SK hynix (memory media), and Penguin Solutions and MemVerge (systems and software).

The Bottleneck: Compute outran memory by 600x

The mechanics start with a scaling divergence. Over the past two decades, peak compute (FLOPS) has grown roughly 3.0× every two years — about 60,000× cumulatively — while DRAM bandwidth grew ~1.6× per two years (~100×) and interconnect bandwidth ~1.4× (~30×). That is a ~600× gap between how fast processors can compute and how fast memory can feed them. This is the “memory wall,” and it is the single largest cost driver in LLM inference today.

As I detailed in Why KV Cache and Memory Drive AI Economics, a 128K-token conversation on a 70B-parameter model can consume ~40GB of HBM per concurrent user — memory that scales linearly with users and context length, unlike the fixed footprint of model weights. HBM is the fastest tier, but it is capacity-constrained and priced at a premium: a single HBM3E stack runs $60–100 against $5–10 for an equivalent slice of DDR5, and HBM consumes roughly 3× the wafer area per gigabyte. The result in 2026 is a genuine DRAM shortage. CXL exists to put a coherent, low-latency tier between scarce HBM and a slow storage hop — and to recover the DRAM that is already installed but stranded.

The Demand Driver: Context windows are the underappreciated trend

The demand curve underneath the memory wall is context length. Two years ago GPT-4 shipped with an 8K-token window; today Gemini runs at 2M and Claude- and GPT-5-class models sit at 1M+, with roughly thirteen frontier models now shipping million-token windows. Micron EVP Manish Bhatia told JPMorgan in 2026 that context windows are expanding ~30× per year. Every token in a window generates KV-cache data that must live in memory during inference, so a single 1M-token session can consume hundreds of gigabytes of KV cache per user — and agentic workflows, where one request fans out into many chained model calls each carrying its own context, make that growth combinatorial rather than linear.

This is the flywheel: longer context → larger KV caches → more HBM and DRAM consumed per session (a direct tailwind for Micron and the NAND makers), and, when the cache overflows on-package memory, a forced spill outward to disaggregated CXL memory (where Penguin Solutions and the controller vendors monetize). A GPU’s on-package HBM is physically bounded by interposer area and stack height to roughly 80–192 GB; once the working set exceeds that, the only question is which tier catches the overflow.

How CXL works, in plain terms

Strip away the jargon and CXL does one thing: it lets a CPU reach extra memory sitting on a PCIe link and use it almost like its own RAM. It does this with three sub-protocols that share a single physical wire. CXL.io handles discovery and configuration — the housekeeping, functionally identical to PCIe. CXL.cache lets a device read the host’s memory while staying coherent (kept in sync). And CXL.mem — the important one — lets the host read the device’s memory with ordinary load/store instructions, no storage driver in the path.

CXL.io → find & set up the device · like PCIe
CXL.cache → device reads the CPU's memory, in sync · coherent
CXL.mem → CPU reads the device's memory, load/store · the key one

The practical consequence is the whole point of the technology: software treats CXL-attached DRAM the same way it treats local DRAM, just ~200 nanoseconds further away (about a NUMA hop), versus the ~100 microseconds it would take to reach an SSD. Around this, the spec defines three device types: Type 1 (an accelerator or SmartNIC with no memory of its own), Type 2 (a GPU or FPGA that has its own memory and also shares the host’s), and Type 3 (a plain memory expander or pool — the workhorse deployed against the memory wall).

Memory semantics, not I/O semantics, is what makes CXL a memory technology rather than just a faster peripheral bus.

The Roadmap: From expansion slot to memory fabric

Each spec revision pushed CXL further from “a card in a server” toward “composable memory across a rack.” CXL 1.1 (2019) did direct-attached expansion on a PCIe 5.0 PHY at 32 GT/s. CXL 2.0 (2020) added switching and memory pooling, plus hot-plug, QoS, and link encryption. CXL 3.0 (2022) moved to a PCIe 6.0 PHY at 64 GT/s with PAM-4 signaling, doubled bandwidth, and introduced fabrics: multi-level switching, memory sharing across hosts, peer-to-peer DMA, and addressing for up to 4,096 nodes. CXL 3.1 (2023) added multi-tier switching and trusted-execution security. And CXL 4.0, released in November 2025, doubled per-lane bandwidth again on a PCIe 7.0 PHY at 128 GT/s, with bundled ports and multi-rack pooling. The throughline: every generation widened the blast radius of a single memory allocation, from one motherboard to an entire rack.

Who is behind CXL — and what CXL is not

CXL is governed by the CXL Consortium, founded in 2019 by Alibaba, AMD, Cisco, Dell EMC, Meta, Google, HPE, Huawei, Intel, and Microsoft — a roster that signals hyperscaler-driven standardization rather than one vendor’s proprietary play. Critically, the Consortium then won the interconnect wars: it absorbed Gen-Z (2021), OpenCAPI (2022), and the CCIX effort, consolidating a fragmented landscape into one standard with one roadmap. That removed the standards-war risk that historically discourages capex on new interconnect silicon, which is why every major CPU — Intel Xeon, AMD EPYC, and the Arm server ecosystem — now ships CXL root-port support natively, and why AMD, Nvidia, and Samsung all sit on the board.

One clarification for the technical reader: CXL is not the GPU-to-GPU scale-up fabric. Its bandwidth and radix are wrong for that job, which belongs to NVLink (proprietary) and UALink (open). CXL won the CPU-to-memory layer; it never seriously contested the accelerator-to-accelerator layer. The two are complementary — and the most telling signal is that even Nvidia supports CXL memory expansion on Grace and Blackwell while building NVLink for scale-up.

Key Players

Connectivity silicon: Astera, Credo, Marvell

Astera Labs (ALAB) is the highest-profile pure-play. It IPO’d in March 2024 at more than 50× trailing revenue — I broke down the filing in Astera Labs IPO: S-1 Teardown, when Aries retimers drove 90%+ of revenue. The scale since is the story: full-year 2025 revenue hit $852.5M (+115%), and the most recent quarter — Q1 FY2026 (reported May 2026) — set a record at $308.4M, up 93% year-over-year and 14% sequentially, at a 76.4% non-GAAP gross margin and a 36.2% non-GAAP operating margin, with PCIe Gen 6 already more than a third of revenue. Leo, its CXL smart-memory-controller line (CXL 2.0, up to 2TB per controller), is the strategic bet — it powers Microsoft Azure’s M-series VMs in the industry’s first announced CXL-attached memory deployment, and in distributed inference with Penguin Solutions it has shown 75% higher GPU utilization and 2× inference throughput, plus 40% faster time-to-insights and up to 3× concurrent LLM instances for RAG. Leo plugs into vLLM, TensorRT-LLM, and SGLang with no application changes.

Credo (CRDO) occupies the adjacent retimer and SerDes layer. The growth has been violent: fiscal 2026 revenue hit $1.3B, up 206% year-over-year, with a record Q4 of $437M (+157% YoY) at ~68% gross margins — and the stock now carries a roughly $50B market cap, four times the level when I first covered it in From Niche to Necessary: Credo’s Quiet Domination of AI Interconnects. Its retimers support PCIe 5.0/6.0/7.0 and CXL interchangeably, so it captures CXL-attach revenue regardless of which controller or memory vendor a hyperscaler standardizes on. Marvell, meanwhile, is building the switch and fabric layer underneath both. Its Structera A near-memory accelerators (16 Arm Neoverse V2 cores, up to 200 GB/s and 4TB) and Structera X expansion controllers were the first with four memory channels and inline compression on 5nm; then in March 2026 it announced the Structera S 30260, a 260-lane CXL 3.0 switch delivering up to 4 TB/s of aggregate bandwidth for rack-level pooling, sampling Q3 2026 — built on Marvell’s $540M acquisition of XConn, closed February 2026, the largest deal in CXL switching to date.

Memory and systems: Micron, Samsung, SK hynix, Penguin

The memory makers supply the capacity the connectivity silicon pools, and CXL is their channel into sockets that are otherwise capacity-capped. Micron (MU) is the cleanest read on the demand: fiscal Q2 2026 revenue hit $23.86B, up 196% year-over-year, at a ~74% gross margin, with Q3 guided near 81% and management describing HBM and high-capacity DRAM as effectively sold out ahead of supply. Its CZ120 CXL modules ship in 128GB and 256GB capacities over PCIe Gen5 x8 at ~36 GB/s — within a hair of a DDR5-4800 RDIMM’s 38.4 GB/s. Samsung’s CMM-D line reached an industry-first 512GB module, and SK hynix ships its 96GB CMM-DDR5 with pooling support. None of them needs CXL to sell DRAM, but all of them need a way to sell more DRAM per core while HBM eats fab allocation — and NAND makers (Micron again, plus SanDisk) pick up the colder KV-cache tiers below CXL.

At the top of the stack, Penguin Solutions (PENG) is the clearest evidence that CXL demand is now inference-driven. In March 2026 it shipped what it calls the industry’s first production-ready CXL-based KV cache server, the MemoryAI appliance — 3TB of DDR5 plus up to eight 1TB CXL add-in cards for up to 11TB total — purpose-built to offload KV cache from GPU HBM, with roughly 10× faster access than NVMe-based approaches and out-of-the-box compatibility with NVIDIA’s Dynamo framework. Penguin frames the workload bluntly: inference is roughly 30% compute and 70% memory. The pull is showing up in results — Q2 FY2026 integrated-memory sales rose 63% year-over-year to $172M, and the company raised full-year guidance to ~12% growth.

Why CXL Now?: The 2026 memory squeeze

The reason all of this is happening at once is a supply shock. HBM has reallocated wafer capacity away from conventional DRAM — it consumes roughly 3× the wafer area per gigabyte — sending DRAM prices up about 90% quarter-over-quarter in Q1 2026 and leaving hyperscalers short on the cheap memory they used to take for granted. As TechInsights’ James Sanders put it alongside Marvell’s switch launch, limited DRAM availability and runaway pricing are “upending the plans of data center operators.” CXL is one of the few near-term levers: it recovers DRAM that would otherwise sit stranded (up to 25% of installed memory at peak, per Microsoft’s Azure data), and it lets operators keep older, cheaper memory generations in productive service instead of paying spot prices for the newest DDR5.

Each generation of Nvidia GPUs adds more HBM capacity and bandwidth. Yet there are fundamental packaging and physical constraints on how much HBM can be integrated with a single GPU, limiting how far this scaling can continue.

What this means — and what breaks

CXL’s economics favor the infrastructure layer over any single silicon architecture. Astera and Credo monetize per-lane regardless of which CPU or accelerator sits on either end — a more durable position than betting on a GPU generation. Marvell’s switch converts pooling from a server feature into a rack-level fabric decision, raising both contract size and switching cost once a hyperscaler commits to a topology. The memory vendors get a second demand channel that requires no new fab, exactly when HBM allocation is tight. The risk to the thesis is timing, not direction. CXL-attached memory carries ~200ns of added latency — immaterial for KV cache offload and pooling, but disqualifying for the most latency-sensitive accelerator-local work, which keeps HBM taking share at the high end. And the disaggregation dividend is real but bounded: Microsoft’s Pond study found pooling across 8–16 sockets recovers ~7% of DRAM (hundreds of millions of dollars at cloud scale) against stranding that can hit 25% — meaningful, but not a panacea. That ceiling is what determines how much of the memory wall CXL actually absorbs versus HBM growth and software-side efficiency.

Summary

CXL solves a specific, quantifiable problem: memory bandwidth and capacity have scaled ~600× slower than compute over two decades, and AI inference — KV cache in particular — drives per-user memory consumption that fixed-memory architectures cannot absorb. Governance by a hyperscaler-led consortium that swallowed its two main rivals removed the standards-war risk that usually slows adoption, and four spec revisions since 2019 have carried CXL from single-server expansion to multi-rack fabric. Astera Labs and Credo monetize the connectivity layer, Marvell is building the switch fabric, Micron, Samsung, and SK hynix supply the capacity, and Penguin Solutions is shipping inference-specific appliances on top. Yole expects CXL-attached memory to exceed $20B by 2030 and CXL to carry ~31% of server DRAM bits by 2028; the CXL component market alone is forecast to grow from ~$1.9B in 2024 to ~$12.3B by 2030 at a ~32% CAGR. The next 12–18 months — through Marvell’s Structera S sampling and the first wave of CXL 4.0 silicon, against the worst DRAM shortage in years — will determine how much of the memory wall gets absorbed by CXL versus solved with HBM and smarter software instead.

In Q2 FY2026 alone, Broadcom booked more than $30 billion in AI semiconductor orders against $10.8 billion in shipments — a 3:1 book-to-ship ratio that means the backlog is growing nearly three times faster than revenue. The company carries a $73 billion contracted AI chip backlog ($53 billion of which is custom accelerators) and is guiding to more than $100 billion in AI semiconductor revenue for FY2027. At Bloomberg Tech on June 5, Hock Tan described the demand environment simply: “insatiable growth.” We first covered Broadcom’s XPU strategy in depth in 2024 — what has changed since is the scale. This piece is the full update.

From HP’s Lab to AI Infrastructure Backbone: M&A and the “Hock Tan Playbook”

Broadcom’s lineage starts in 1961, when Hewlett-Packard founded a semiconductor laboratory in Palo Alto to supply components for its own test and measurement equipment. That division spun out as part of Agilent Technologies in 1999, then went private in 2005 when KKR and Silver Lake bought Agilent’s Semiconductor Products Group for $2.66 billion — 6,500 employees, manufacturing in Singapore, leading positions in wireless filters, fiber optics, and industrial motion control. Hock Tan, who had just sold Integrated Circuit Systems to IDT, was installed as CEO in 2006.

Avago went public on Nasdaq in August 2009 (ticker: AVGO) at roughly a $2.5 billion market cap. Over the following 14 years, Tan ran six major acquisitions that together created the company carrying the Broadcom name today:

1. LSI Corporation ($6.6B, 2013)

2. Broadcom Corporation ($37B, 2016, with Tan adopting the acquired company’s name)

3. Brocade ($5.5B, 2017)

4. CA Technologies ($18.9B, 2018)

5. Symantec Enterprise ($10.7B, 2019)

6. VMware ($69B, November 2023).

Total acquisition capital deployed: over $140 billion. A separate $117 billion hostile bid for Qualcomm was blocked by President Trump in March 2018 on national security grounds.

The logic underlying each deal is consistent — what observers call the “Hock Tan playbook.” Find franchise assets with dominant market share and customers that cannot switch without 12–24 months of re-certification. After close: cut lower-priority R&D, eliminate duplicative sales and marketing overhead, run each of 24 divisions as a fully independent P&L.

The approach generated the loudest backlash of Tan’s career when applied to VMware — some enterprise customers reported license cost increases of 150% to 1,000% as Broadcom collapsed 8,000+ SKUs into four subscription bundles. The playbook worked anyway: VMware software now runs at 93% gross margin. What Tan said at Bloomberg Tech on June 5, 2026 is the most important thing to understand about where Broadcom is today: he has stopped prioritizing M&A, because AI organic growth — $30B booked in a single quarter — is better than any deal he can buy. That is a different kind of statement from a CEO who built a $1.7 trillion company almost entirely through acquisitions.

Financial Overview

Broadcom’s FY2025 non-GAAP gross margin was 78.6% on $63.9B of revenue — $43.0B adjusted EBITDA (67.3% margin), $26.9B free cash flow. Software-company margins on chip revenue, held by 18,000 patents and $11.0B of annual R&D across 24 product divisions. Q2 FY2026 reset the baseline: total revenue $22.2B (+48% YoY), AI semiconductor revenue $10.8B (+143% YoY), EBITDA $15.2B at 69% — both records. Annualized AI run rate: $43.2B. Q3 FY2026 guidance accelerates to $29.4B total (+84% YoY) and $16.0B AI semiconductor (+200% YoY) — a $64B annualized AI run rate and $117.6B total annualized. Gross margin guides to ~74%, down from 78.6%, entirely mix-driven as higher-volume AI silicon displaces legacy products — not structural compression. $30B in Q2 bookings against $10.8B shipped: demand is running 3:1 ahead of supply. The constraint is capacity, not orders.

Custom Silicon: XPUs for Hyperscalers

The most valuable thing Broadcom does right now is also one of the least visible to outsiders. It co-designs custom AI accelerators — internally called XPUs — for the largest compute buyers in the world. The process is collaborative and long: Broadcom embeds engineers with a customer’s ML infrastructure team, sometimes for years, and produces a chip tuned to that customer’s specific model architecture, memory bandwidth requirements, and power envelope. The resulting silicon typically outperforms a general-purpose NVIDIA GPU on the target workload by a meaningful margin and at 30–50% lower total cost of ownership, because it is not carrying the overhead of generality. The customer gets a better chip for their specific task. Broadcom gets a multi-year supply contract, premium ASP for the design work, and a relationship that is structurally difficult to exit — because migrating a chip design to a different design house takes 2–3 years and a different foundry partnership can take longer. The company has confirmed six active XPU customers: Google, Meta, OpenAI, Anthropic, ByteDance, and Fujitsu — all on multi-year contracts for what management is describing as “multi-gigawatt” compute deployments.

Google is the longest-standing relationship — seven generations of co-designed TPUs since 2014. The 7th-generation “Ironwood” TPU, built on TSMC 3nm with a dual-chiplet design co-developed with both Broadcom and MediaTek, delivers 4,614 FP8 TFLOPS with 192 GB HBM3E. Ironwood deploys in 9,216-chip pods delivering 42.5 FP8 exaflops of aggregate compute at a TCO Google claims is 44% below an equivalent NVIDIA GB200 server. Meta extended its MTIA partnership in April 2026 through 2029, targeting the industry’s first 2nm AI compute accelerator — integrating four N2 compute dies, one I/O die, and six HBM stacks in a single CoWoS-S package. Meta has publicly committed to “more than 1 gigawatt of custom silicon to start, then multiple gigawatts over time,” implying $30–50B+ of procurement over the contract period at current silicon ASPs.

OpenAI signed a multi-year co-design agreement with Broadcom in October 2025, valued at approximately $10 billion, to produce its “Titan” XPU — a chip internally targeting a 90% reduction in inference cost versus general-purpose GPUs. Titan is fabricated on TSMC N3 with HBM3E, led by a 40-person in-house chip team under VP of Hardware Richard Ho (ex-Google). Mass production is targeted for the second half of 2026; a second generation on TSMC’s A16 (1.6nm) node is in planning, paired with Samsung HBM4 under a separate memory supply agreement. The chip is for internal inference use only, not commercial sale. Anthropic was revealed as Broadcom’s previously undisclosed “$10 billion customer” in December 2025. Anthropic has committed to a $21 billion chip relationship with Broadcom while simultaneously deploying up to one million Google TPU v7 Ironwood units — the largest deal in Google Cloud history — reflecting a multi-vendor compute strategy as its annualized revenue has grown from ~$9 billion in late 2025 to over $30 billion in 2026.

The $73 billion AI backlog — equivalent to roughly 1.5 years of total company revenue — reflects contracted demand across these six design wins. XPU design cycles typically lock in 3–5 year supply commitments, as migrating chip designs between foundry partners and design houses is a 2–3 year engineering project. Broadcom disclosed at Q2 FY2026 earnings that it has booked an additional $6 billion in AI orders from two new customers not previously included in backlog guidance, suggesting the disclosed six-customer figure may expand further by year-end.

The packaging technology underpinning the next generation of XPUs is called 3.5D XDSiP (Extended Die System-in-Package). A single die is constrained by the lithography reticle limit — roughly 800mm². XDSiP breaks that ceiling by bonding multiple compute dies face-to-face via TSMC’s SoIC process, combined with 2.5D CoWoS interposer integration — yielding a single package with up to 6,000mm² of silicon and 12 HBM stacks. The first production XDSiP product shipped in February 2026: four compute dies, one I/O die, six HBM stacks. More than five products follow. This capability is co-developed with TSMC using bonding processes unavailable through any alternative foundry. A competitor cannot replicate it by hiring packaging engineers — it requires years of joint process development on the same nodes. No one catches up in a single tape-out cycle.

On June 9, 2026, the AI XPV Platform was announced — a $35 billion first tranche led by Apollo Global with Blackstone participation to build hyperscale AI data centers anchored by Broadcom-designed XPUs, targeting 20 gigawatts of capacity by 2028 across Fluidstack-operated sites. Anthropic is the anchor tenant, deploying a 1 GW XPU cluster starting mid-2026. What makes this announcement significant is not the dollar figure — it is what the structure reveals about how the AI capital stack is vertically integrating. Anthropic co-designed its chips with Broadcom. Those chips will run in data centers financed by Apollo and Blackstone. The data centers will be operated by Fluidstack. The silicon is fabricated by TSMC. Broadcom sits at the design layer connecting all of it. A foundation model lab that co-designs its inference chip with Broadcom, then deploys it in a privately financed data center built around that chip, has made a 5–7 year infrastructure commitment. That is not a vendor relationship. It is a structural dependency — and it runs in Broadcom’s favor at every layer.

The Competitive Landscape: Marvell, MediaTek, and NVIDIA

Broadcom and Marvell together control an estimated 95% of the custom AI ASIC co-design market. Marvell is the clear number-two — it co-designs Amazon Trainium and Microsoft Maia, expects AI data center revenue of approximately $11 billion in FY2026 (up from ~$4B the prior year), and had more than 50 new design opportunities across 10+ customers as of early 2026. MediaTek is the less obvious threat: it co-designed Google’s TPU v7 Ironwood alongside Broadcom, targets $2B in AI ASIC revenue for 2026, and is pushing 448G SerDes development ahead of any competitor’s current 224G production. The more direct risk from MediaTek is not a frontal attack — it is Google continuing to diversify its silicon partners across future TPU generations, quietly eroding Broadcom’s share of the most important single XPU relationship. On Ethernet switching, Marvell’s Teralynx T100 — 102.4 Tbps, TSMC 3nm — began sampling in Q2 2026, with volume production targeted mid-2027. Broadcom is already shipping Tomahawk 6 at volume. The lead is 12–18 months on the exact refresh cycle that is happening now.

The sharper version of the bear case is internalization: the possibility that hyperscalers build chip design capability until they no longer need a co-design partner. Amazon has already done this — Trainium runs through Annapurna Labs, an internal design house acquired in 2015. Microsoft’s Maia 200 (140 billion transistors, TSMC 3nm, 216 GB HBM3E) is a fully internal Azure design. Google has the deepest internal chip team of any hyperscaler. The honest read: Google, Amazon, and Microsoft use Broadcom not because they lack chip designers — they have thousands — but because Broadcom has SerDes IP, 3.5D advanced packaging expertise, and production ramp credibility that takes years to build internally. Each XPU generation starts planning 3–5 years before tape-out. The $73B backlog is decisions already made. Whatever a customer decides to do differently in 2030 does not move revenue in the window between now and then.

NVIDIA is the most structurally interesting threat — not as a co-design competitor but as a direct attack on the economic rationale for custom silicon itself. At SC25 in November 2025, NVIDIA introduced Vera Rubin with a public claim of 90% lower AI running costs versus Blackwell. If that holds in production, it narrows the TCO gap that makes a custom XPU worth a 3-year co-design commitment. The ASIC value proposition has historically been 30–50% better TCO on a target workload. NVIDIA is explicitly targeting that number. Analysts project Broadcom will hold approximately 60% of the custom AI processor market by 2027 — but that projection is partly conditional on Vera Rubin underdelivering at scale, as Blackwell underdelivered on its initial dense-deployment training throughput claims. The structural counter: inference workloads are more fixed and more amenable to hardware specialization than training. The inference-focused ASIC pipeline — OpenAI’s Titan chip, Anthropic’s contracted chips — was not commissioned because anyone expected Vera Rubin to fail. It was commissioned because inference economics favor specialization independent of what NVIDIA’s next GPU delivers.

Ethernet Networking Silicon: The Open Standard Play for AI

A 100,000-GPU cluster on InfiniBand requires NVIDIA’s Quantum switches, NDR/XDR cables, and ConnectX NICs at every layer — a single-vendor supply chain with a single vendor’s pricing power. Broadcom’s counter-positioning is simple: open. Any OEM chassis, any standards-compliant transceiver, any RDMA-capable NIC. Broadcom supplies the dominant switch ASIC at the center and takes no position on what surrounds it. The market validation is visible at every level: Google’s Ironwood TPU clusters run Ethernet. Meta’s entire RoCE fabric is Ethernet. Microsoft Azure has diversified newer GPU cluster builds away from InfiniBand. And Arista Networks — the dominant hyperscale and enterprise Ethernet switch vendor — builds its AI spine products almost entirely on Broadcom Tomahawk ASICs; Arista’s revenue grew 27% YoY in Q1 FY2026, driven largely by AI cluster switching deployments that run on Broadcom silicon. Each deployment reinforces the reference architecture for the enterprise builds that follow. Broadcom holds an estimated 60–70% merchant switch ASIC share at the center of that architecture, built over a decade against challengers that kept arriving and failing (Innovium, Pensando, Intel Barefoot). No competitor produces 106 Gbps PAM4 SerDes at production yield without years of investment. That gap does not close in a product cycle.

Broadband and Wireless: The Installed-Base Business

The $4.2B non-AI semiconductor segment annualizes at roughly $17 billion — a business that would be a top-20 chipmaker on its own, but is structurally overshadowed by the AI growth rates surrounding it. It is not a growth business. It is a franchise business: high share, high switching costs, low competitive risk. Changing the SoC in a cable modem gateway requires re-certifying with every broadband operator, a process that takes 18–24 months. Nobody does it casually.

Broadcom’s BCM33xx DOCSIS 4.0 SoCs are the primary silicon in the majority of residential and SOHO gateways globally — Comcast, Charter, and Liberty Global are all in active DOCSIS 4.0 deployment. Gateway SoC ASP is $15–25 across a 200M+ global subscriber base; thin per unit, stable in aggregate. Fiber/PON follows the same pattern: XGS-PON chipsets for 10G passive optical networks ship into Tier-1 telco fiber-to-home deployments across North America, Europe, and Asia Pacific. On WiFi, Broadcom and Qualcomm hold an estimated 80% combined share of merchant access-point silicon; second-generation WiFi 7 chips entered volume production in 2026 with 3-link Multi-Link Operation reducing latency ~50% versus prior-generation implementations. None of these markets will 4x in two years. All of them will still exist in 10 years, with Broadcom inside them.

Storage Connectivity and Industrial Semiconductors

Storage connectivity is the least glamorous segment in Broadcom’s portfolio — and one of the most strategically useful to understand, because it illustrates how the company thinks about market position more broadly. MegaRAID controllers and SAS host bus adapters are not growing markets. The companies that buy them — enterprise IT departments, financial services firms, healthcare networks — are not accelerating their storage refreshes. But every server rack that has a MegaRAID controller needs a MegaRAID controller when it fails or refreshes, and the qualification cycle to introduce a competing product into that environment is 12–18 months minimum. Broadcom does not need the market to grow. It needs the installed base to persist. It will.

The strategically interesting product in this segment for AI is the PCIe switch (PEX series). As AI server chassis disaggregate — compute, memory, NIC, and storage as separate modules on a shared PCIe fabric — PCIe switching silicon becomes the internal interconnect that determines bandwidth and latency within a chassis. PCIe 6.0, now in qualification at 64 GT/s, doubles per-lane bandwidth versus PCIe 5.0. Broadcom is the dominant PCIe switch supplier across enterprise and hyperscale; this is no longer a storage story, it is an AI server architecture story. The segment contributes an estimated $600–800M per quarter. The Brocade-derived Fibre Channel portfolio serves financial services, healthcare, and government SAN environments — a declining market generating durable replacement revenue through 2027. Quiet, durable, not going anywhere.

Infrastructure Software: VMware and Beyond

Broadcom acquired VMware in November 2023 for $69 billion. The software segment generated $7.2B in Q2 FY2026 (annualized: $29B, +9% YoY) and encompasses VMware’s virtualization and private cloud stack (vSphere, vSAN, NSX, Aria), the Carbon Black security platform, and the Symantec enterprise security portfolio. The Q2 FY2026 gross margin on the software segment came in at 93% — among the highest reported by any business unit of a major diversified technology company. Annual Recurring Revenue is growing at +17% YoY, with the vast majority of VMware’s installed base now migrated from perpetual licensing to subscription bundles under VCF (VMware Cloud Foundation). VCF 9.1, released in 2026, added native support for GPU and heterogeneous compute orchestration, allowing vSphere clusters to directly provision and manage GPU workloads alongside traditional CPU VMs — a direct response to enterprise hybrid AI infrastructure demands. When asked at Bloomberg Tech on June 5 whether generative AI posed a risk to VMware’s software lock-in, Hock Tan was unambiguous: “We are not seeing it.” Enterprise data centers running VMware-managed HCI are adding AI inference workloads as an additive layer, not replacing the underlying virtualization stack. The strategic logic remains what it has always been: VMware’s private cloud stack runs on hyperconverged infrastructure in the same data centers as Broadcom’s networking chips — account control at both the software and silicon layer, across the same Fortune 500 customer base.

Revenue Trajectory and 2027 Outlook

“Customers are coming to us for Opus 4.7, ChatGPT 5.5, Gemini 3.5 — they’re naming the models and contracting compute now. That’s the insatiable growth we’re seeing.”

— Hock Tan, CEO, Bloomberg Technology, June 5, 2026

In FY2024, Broadcom’s AI semiconductor revenue was approximately $12 billion for the full year. In FY2026, the company is guiding to $56 billion — a 4.7× increase in two years. FY2027 guidance, reiterated at Q2 FY2026 earnings on June 3, is in excess of $100 billion. The single most revealing data point in Q2 FY2026 is not the revenue print but what sits behind it: Broadcom disclosed more than $30 billion in new AI orders booked in the quarter against $10.8 billion shipped. A 3:1 book-to-ship ratio means the backlog is expanding nearly three times faster than revenue. Management disclosed an additional $6 billion from two new customers not previously in backlog figures. The $73B number — $53B custom accelerators, the remainder networking — is a floor, and the floor is rising. What Tan described at Bloomberg Tech is the mechanism: hyperscalers are contracting compute capacity against future model versions not yet in production, with 2–3 year lead times baked in. The chip decisions being made today are for models that will run in 2028.

Broadcom’s FY2025 non-GAAP gross margin was 78.6% on $63.9B of revenue — $43.0B adjusted EBITDA (67.3% margin), $26.9B free cash flow. Software-company margins on chip revenue, held by 18,000 patents and $11.0B of annual R&D across 24 product divisions. Q2 FY2026 reset the baseline: total revenue $22.2B (+48% YoY), AI semiconductor revenue $10.8B (+143% YoY), EBITDA $15.2B at 69% — both records. Q3 FY2026 guidance accelerates to $29.4B total (+84% YoY) and $16.0B AI semiconductor (+200% YoY) — a $64B annualized AI run rate and $117.6B total annualized. Gross margin guides to ~74%, down from 78.6%, entirely mix-driven as higher-volume AI silicon displaces legacy products — not structural compression.

At a ~$1.7 trillion market cap as of June 2026, AVGO trades at 14.5× annualized forward revenue and ~25× run-rate FCF — implying a ~4% FCF yield (~$67B annualized FCF, based on ~57% FCF margins from Q2 non-GAAP operating data applied to Q3 guidance). At 4%, the yield is defensible for a business with 18,000 patents, 93% software gross margins, and a $73B contracted backlog — but it assumes that backlog converts on schedule. The bear case is a 12–18 month delay in XPU shipment ramps from packaging yield failures or TSMC allocation constraints; at 25× FCF, that reprices the stock, not the business. The bull case is that $30B in quarterly bookings is early-cycle — that the eventual compute buildout is larger than the current backlog implies, and the next $73B is already forming.

The structural risks are known and priced. Google almost certainly represents 20%+ of total company revenue — customer concentration is the primary earnings risk. Broadcom has 100% foundry exposure to TSMC; any disruption to the Taiwan supply chain propagates immediately. None of these risks have a near-term catalyst. The contracted backlog is locked. The $30B quarterly booking rate, with two new undisclosed customers, suggests the demand curve is not flattening.

Sources

• Broadcom Q2 FY2026 Earnings Press Release (PR Newswire)

• Broadcom Q1 FY2026 Financial Results (investors.broadcom.com)

• Futurum Group: Broadcom Q2 FY2026 Analysis

• Tom's Hardware: Custom AI ASIC State of Play — May 2026

• NADDOD: Tomahawk 6 Deep Dive

• The Register: Broadcom Takes a Tomahawk to NVIDIA's AI Networking Empire

• ServeTheHome: Tomahawk 6 Launch

• Packet Pushers: Jericho3-AI Ethernet Fabric

• Manufacturing Dive: Broadcom–Meta 2nm Partnership (April 2026)

• Broadcom: Second-Generation WiFi 7 Chips

• Tech Insider: Broadcom Q2 2026 AI Revenue

• Prime Trading: AVGO Q1 FY2026 Deep Dive

• CNBC: Broadcom Reveals Mystery $10B Customer is Anthropic (Dec 2025)

• TrendForce: OpenAI Titan Chip on TSMC N3, Second-Gen on A16 (Jan 2026)

• Broadcom Q2 FY2026 Earnings Call Transcript (June 3, 2026)

• Tom's Hardware: The Custom AI ASIC State of Play, May 2026

• MLQ News: Marvell Teralynx T100 Launch

• TrendForce: MediaTek Doubles 2026 AI ASIC Target to $2B (Apr 2026)

• Broadcom: Jericho4 Shipping — Enabling Distributed AI Computing (Aug 2025)

• Business Chief: Hock Tan Leadership Profile

• NetworkTigers: A History of Broadcom — HP Origins to VMware and AI

• NPR: Trump Blocks Broadcom Takeover Bid for Qualcomm (Mar 2018)

• Redress Compliance: Broadcom VMware Pricing Report 2026

A semiconductor die cannot connect directly to a circuit board: voltage levels are incompatible, I/O pitch differs by three orders of magnitude, and thermal density would destroy the board. The substrate bridges that gap. It is constructed from alternating layers of copper conductors and ABF dielectric film — and without it, Nvidia cannot ship an H100, AMD cannot ship an MI300X, and Google cannot finish a TPU.

How a food company became AI’s silent gatekeeper

Ajinomoto Co.’s primary revenue is still food seasonings. But in 1996, Intel commissioned its chemical division to develop a film-type insulator for semiconductor packaging. Ajinomoto’s team — leveraging amino acid chemistry — delivered a working formulation in four months. Mass production started in 1999.

ABF has since become structurally irreplaceable. No credible alternative exists at production scale. Mitsubishi Gas Chemical and Sekisui Chemical have design wins in lower-performance applications, but neither has qualified at the layer counts or yields required for AI accelerators. Ajinomoto’s Fine-Techno subsidiary controls approximately 95% of the CPU/GPU-grade ABF market — a number that has not moved in a decade.

The moat is functional lock-in, not patents. Qualifying a new ABF supplier at a top-tier substrate manufacturer takes 18–24 months minimum. Even if a competitor produced an equivalent film today, no substrate maker could switch without halting production for Nvidia, AMD, or Intel.

Why substrates matter more than ever

The substrate was a low-drama component. A CPU in 2015 needed 4–6 ABF layers, fit in a ~40mm package, cost $40–55, and delivered in 6–8 weeks. Three structural shifts changed this simultaneously — and none reverses.

Transistor scaling economics degraded. Moving 5nm→3nm costs 2–3× more per transistor than the prior node. The industry’s answer is chiplets: packing multiple dies from different process nodes into a single package. Every chiplet design requires a substrate, and it always needs more ABF layers than the monolithic die it replaced.

Package size and complexity exploded. An H100 GPU requires 12+ ABF layers in a package exceeding 120×120mm. Blackwell’s B200 uses CoWoS-L with a die footprint above 800mm² — among the largest packages in production history. ABF content per AI accelerator is roughly 3–4× what a CPU required five years ago.

HBM adds routing complexity. A B200 carries eight HBM3e stacks adjacent to the compute die. The substrate must fan out connections for all of them simultaneously. Each additional HBM stack adds layers, footprint, and ABF content. Hyperscale operators installed more than 1.2 million AI accelerator units in 2025.

$5.8B today, $11.7B by 2030

Pricing tells the story as clearly as volume. ABF substrate ASPs rose from ~$65 (2024) to ~$82 (2025) — a 26% increase in twelve months. Lead times for 14-layer substrates peaked at 28 weeks in early 2025. Mid-tier fabless designers either paid spot premiums of 15–20% above contract or delayed tape-outs by one to three quarters.

Five companies, 82% of the market

The high-end FC-BGA substrate market is split among five manufacturers: Ibiden (~35% share), Shinko Electric (~18%), Unimicron (~14%), AT&S (~10%), and Nan Ya PCB (~5%), with the remainder fragmented across smaller Asian suppliers. All five are capacity-constrained and all five have announced expansions — which obscures the more relevant fact: lead times of 20–28 weeks indicate demand is outrunning committed buildout, not just announced intentions. Shinko, majority-owned by Fujitsu, has historically been CPU-concentrated; it is now qualifying for AI workloads, but layer count limitations keep it behind Ibiden for the most complex packages.

AT&S is the only European supplier at meaningful scale and has positioned itself as a geopolitical hedge, receiving EU subsidy support for its Austrian facility — strategic for customers who want non-Asian supply, but it runs at roughly one-fifth of Ibiden’s AI substrate revenue. Unimicron and Nan Ya are price-competitive at lower layer counts but have not demonstrated volume yield at 14+ layers for GPU-class packages. Ibiden’s lead is not just share — it is yield maturity at the layer counts that matter.

The substrate fab

Ibiden’s ¥500B (~$3.2B) capex commitment across FY2026–2028 is the largest single substrate expansion in history. The Ono Plant opened at ~50% building utilization in October 2025. Kawama Plant Cell 6 (¥220B) targets FY2027. Combined goal: 2.5× current capacity by 2028. Demand projections from hyperscalers suggest this may not fully close the gap.

The concentration risk is unhedged: all of Ibiden’s major plants sit in Gifu Prefecture, a seismically active zone. A single facility disruption would propagate directly to AI chip shipment volumes globally. That risk has no public market hedge.

Three paths challenging the organic substrate

The interposer, panel-level packaging, and glass core substrates converge on the same problem — but on different timelines, with different ABF implications.

Glass substrates and the Corning bet

The problem glass solves is warpage. Organic ABF substrates deform above ~120×120mm packages because of CTE mismatch between the film (~30 ppm/°C) and the silicon die (~2.6 ppm/°C). During reflow, the package warps — the “potato chip” effect — causing yield loss that compounds as die size grows. Every generation of AI GPU package pushes toward or past this limit.

Glass CTE can be engineered between 3–9 ppm/°C, far closer to silicon. It enables finer via density, lower signal insertion loss, and better flatness for lithographic patterning. If glass qualifies at volume, it directly displaces ABF as the substrate dielectric — a long-dated but high-conviction risk for Ajinomoto Fine-Techno.

The timeline risk is real. Glass substrates require entirely new manufacturing equipment, process qualification at multiple yield gates, and supply chain infrastructure. Mass production below $500/substrate is not demonstrated at scale. ABF organic remains the production standard through at least 2028, and likely longer for the majority of the market. Corning is the clearest public equity read-through — but the transition is multi-year, capital-intensive, and yield-dependent.

The ABF control point the market underprices

The substrate layer is one of the few places in AI infrastructure where non-semiconductor companies exercise structural leverage over hyperscaler capex. Ibiden can constrain Nvidia’s shipment volumes. Ajinomoto constrains Ibiden.

The market treats substrate manufacturers as contract manufacturers — process-intensive, capital-heavy, margin-thin. That framing made sense when substrates were commodity components. It does not describe a company whose yield curve at 14+ ABF layers determines whether Blackwell ships on schedule.

Ajinomoto Fine-Techno illustrates how durable that leverage is. A food company’s chemical division holds 95% of the most upstream material input in AI compute — not through patents or price, but through an 18–24 month qualification barrier that no substrate maker can absorb without halting production for its largest customers. That is a control point maintained entirely by switching cost.

The pattern repeats across the production chain. The companies that dominate AI market cap are not always the companies that hold structural leverage. The further you trace the physical stack, the more single-point dependencies you find — held by companies that most AI coverage never mentions.

Thesis

The binding constraint on frontier AI has shifted. Through 2023 it was accelerator supply. By 2026 it is the interconnect — and “interconnect” now spans three distinct distance scales at once: chip-to-memory (millimeters), GPU-to-GPU within a rack (meters), and rack-to-rack across a data center hall (tens to hundreds of meters). The fabric moving activations, gradients, KV-cache traffic, and now memory bandwidth itself between tens of thousands of accelerators has to keep all three scales fed simultaneously, fast enough that the silicon never idles waiting on data.

Copper, the default interconnect medium for sixty years of computing, has hit a hard physical limit at the data rates AI clusters now require. A growing field of replacement technologies is racing to fill the gap — silicon photonics, VCSEL-based near-package optics, THz radio-over-wire, and early-stage microLED interconnects among them — and 2026 is the year several of them moved from roadmap slide to shipping product.

Two approaches anchor this piece because they represent the clearest structural bets being made with real capital today: silicon photonics, which converts data to light and is now central to Nvidia’s and Broadcom’s switch roadmaps, and THz radio-over-wire, an approach pioneered by the startup AttoTude (a Wing portfolio company) that drives terahertz-frequency radio signals directly through wire, skipping optical conversion altogether. These are not rivals fighting over a single socket. They sit at different points on the cost, distance, and power curve, and the more durable read — reinforced by how Marvell, Broadcom, and Nvidia are each shipping multiple interconnect formats side by side inside the same switch families — is that several of these approaches will coexist in the AI data center of 2028–2030, often inside the same rack.

The scale of the prize is enormous regardless of which technologies win which sockets. The five largest hyperscalers are on track to spend roughly $725 billion on AI infrastructure in 2026 alone — nearly triple the $256 billion they deployed in 2024 — and networking and interconnect have become a double-digit percentage of total cluster cost rather than a rounding error. Nvidia has put a number on how seriously it takes this: Nvidia has directed roughly $4 billion of capital specifically into optical component suppliers — split into $2 billion stakes in Lumentum and Coherent, paired with multibillion-dollar purchase commitments, future capacity-access rights, and funding for new U.S.-based fabrication facilities, a direct, dollar-denominated bet that the interconnect layer — not the GPU itself — is now the gating factor on how fast it can ship usable AI compute. McKinsey, separately, estimates that EML laser chips will run 40–60% undersupplied through 2027.

The companies that control the bottlenecks in this build-out — manufacturing capacity, component and architecture IP, and systems-level co-design with the largest compute buyers — are re-rating from cyclical component suppliers into structural infrastructure plays. That re-rating is visible in the numbers today, not just the roadmaps.

The numbers that frame the opportunity:

• Frontier training clusters in 2026 already run 100,000+ GPUs, with “AI factory” designs targeting millions — and copper’s practical reach collapses as speed rises: Marvell’s own data points put it at roughly 5 meters at 100 Gb/s per lane, ~2.5 meters at 200 Gb/s, and effectively unusable even within a rack at 400 Gb/s.

• Hyperscaler AI infrastructure spend is on pace for ~$725 billion in 2026, up from $256 billion in 2024 — and networking and optical interconnect, now a double-digit percentage of total cluster cost rather than a rounding error, represents on the order of $75–100 billion of that spend this year alone.

• Nvidia has committed roughly $4 billion directly into optical suppliers, and its Spectrum-X Photonics SN6800 switch reaches 409.6 Tb/s across 512 ports at 800 Gb/s each — with co-packaged optics cutting power consumption by up to 3.5x and improving resiliency 10x versus pluggable architectures.

• Marvell’s new Teralynx T100 — the industry’s first 102.4 Tb/s switch built specifically for AI data centers — runs on a 3nm process, draws under 1,000W, and delivers roughly 25% lower power than competing solutions, while shipping in pluggable, co-packaged-copper, and co-packaged-optics configurations side by side.

• The silicon photonics market is estimated at roughly $3–4 billion in 2025–2026, scaling to $10–29 billion by 2030–2034 (roughly a 27–30% CAGR across the major forecasts — Mordor Intelligence, MarketsandMarkets, Precedence Research, and Polaris Market Research all now cluster in the high-20s, with several houses revising estimates upward through 2026 as AI-specific demand outpaces the telecom- and consumer-driven base case those models were originally built on), while the broader next-generation AI interconnect market — spanning photonics, VCSEL/NPO, THz, and microLED approaches — could exceed $100 billion annually within the decade, according to industry estimates cited around OFC 2026.

• AttoTude’s THz radio-over-wire approach reports per-lane speeds up to 448 Gb/s and reach up to 40 meters directly over wire — without an optical conversion stage.

The Copper Wall: Why Electrons Stop Scaling

Copper interconnects move data by modulating voltage on a wire — a mechanism that degrades predictably, and non-linearly, as both frequency and distance rise. Marvell CEO Matt Murphy put concrete numbers on this at his Computex 2026 keynote: copper can carry a signal roughly 5 meters at 100 Gb/s per lane, but that reach falls to about 2.5 meters at 200 Gb/s, and at 400 Gb/s copper can no longer reliably make connections even within a single rack. Each time this threshold moves, Murphy argued, the number of links that must convert to optical increases by at least an order of magnitude — which is what is now driving what he called an “explosive” demand cycle across the optics supply chain, with Taiwan’s manufacturing base already expanding in response.

This is the “copper wall,” and it is a property of physics that worsens, not improves, as model and cluster sizes grow. Beyond roughly three meters of reach, attenuation and heat dissipation make copper a net negative versus the alternatives; at million-GPU scale, the share of total system power and latency consumed by electrical interconnect stops being an afterthought and becomes the dominant design variable.

Every replacement technology in this piece — optical, VCSEL-based, THz-over-wire, or microLED — exists because it sidesteps this specific failure mode, via different physical mechanisms. Photons don’t suffer resistive loss the way electrons do. THz-frequency radio signals can be engineered to traverse wire with substantially less of the attenuation that kills copper at lower frequencies. And the industry’s accelerating search for more than one way around this wall is itself the signal: when an entire supply chain starts hedging across several physical bets simultaneously, that is usually a sign the old approach is genuinely exhausted, not just temporarily expensive.

Why This Matters Specifically for Large Models

Transformer-based models are bottlenecked as much by communication as by computation. Mixture-of-experts architectures, tensor and pipeline parallelism, and disaggregated inference — separating prefill from decode across different hardware pools — all multiply east-west traffic between accelerators relative to the FLOPs actually executed. As clusters scale from thousands to hundreds of thousands of GPUs, the ratio of interconnect bandwidth to compute, not raw FLOPs, increasingly determines effective utilization.

Nvidia has made this an explicit architectural priority. Its Quantum-X InfiniBand switches (early 2026) and Spectrum-X Photonics Ethernet switches (second half of 2026, built on the Spectrum-6 ASIC) integrate co-packaged optics directly onto the switch silicon. The flagship SN6800 reaches 409.6 Tb/s across 512 ports at 800 Gb/s — bandwidth that would be physically and thermally impossible to deliver with discrete pluggable copper or pluggable optical modules at that density. Nvidia states CPO cuts power consumption by up to 3.5x and improves resiliency 10x versus conventional pluggable architectures, with its longer-range roadmap following TSMC’s COUPE packaging program toward 6.4 Tb/s at the board level.

Nvidia’s own architecture diagrams make the structural shift legible at the component level. A traditional pluggable switch routes signal from the switch ASIC, across the substrate, through a port cage, into an external optical transceiver — DSP, externally modulated laser, and all — needing eight separate lasers per 1.6 Tb/s of bandwidth. Spectrum-X Ethernet Photonics collapses that chain: the silicon photonics engine sits directly on the substrate beside the switch ASIC, driven by a continuous-wave laser, and the same 1.6 Tb/s now requires just two lasers — a quarter as many laser sources, connectors, and potential failure points, assembled in far fewer steps.

Crucially, this is no longer a future-tense story. Nvidia’s Vera Rubin platform has already adopted Spectrum-X Ethernet Photonics — the first CPO-based switch to enter mass production — which is the clearest evidence yet that co-packaged optics has crossed from proof-of-concept into actual commercial deployment. Nvidia’s own roadmap materials describe this as “CPO co-invention with ecosystem partners”: the first 1.6T silicon photonics chip built around new micro-ring modulators, the first 3D-stacked photonics engine built on TSMC’s process, high-power high-efficiency lasers, detachable fiber connectors for serviceability, and — the detail that matters most structurally — hundreds of patents licensed out to a roster spanning Coherent, Lumentum, Corning, Fabrinet, Foxconn, Sumitomo Electric, Senko, and TSMC, rather than held closed. That is Nvidia subsidizing and standardizing the supply chain underneath its own roadmap, the same playbook it has used with CUDA and NVLink — turning would-be competitors and suppliers into an ecosystem with a shared interest in its architecture winning.

The largest buyer of AI compute has told the market, in shipping silicon, capital commitments, and licensed IP, that the interconnect layer is no longer a procurement afterthought. It is now co-designed with the compute itself — and as optical connectivity extends further into the server, closer to the chip-to-memory link itself, it opens the door to disaggregating compute, memory, and networking entirely, and operating the data center less like a collection of fixed servers and more like a single, dynamically configurable system.

The Near-Term Bridge: VCSEL-Based Near-Package Optics

Co-packaged optics is the destination, but it is not yet the easiest way to ship volume today — CPO requires new packaging, new yield curves, and new supply chains to mature simultaneously. Broadcom’s answer to that gap is VCSEL-based Near-Package Optics (NPO): optical engines built on vertical-cavity surface-emitting lasers, mounted close to — but not integrated into — the switch package.

The appeal is pragmatic rather than visionary. VCSELs are a mature, high-volume, field-proven technology already manufactured at scale for 100G multi-mode applications, which means NPO can be deployed today on proven supply chains rather than waiting for CPO’s packaging ecosystem to mature. Broadcom’s 3.2T VCSEL-based NPO solution is positioned explicitly as a bridge: lower cost and lower power than legacy pluggables, small enough to preserve architectural flexibility, and good enough on wall-plug efficiency across a wide range of speeds that it doesn’t force operators to make an all-or-nothing bet on CPO timelines. For investors, NPO is worth tracking less as a long-term technology winner and more as a signal of sequencing — it is the architecture the industry is choosing to absorb demand while CPO’s manufacturing base catches up, and the companies supplying it (Broadcom prominently among them) get a multi-year revenue bridge while the bigger structural transition plays out underneath.

Two Complementary Paths Beyond Copper: Photonics and THz Radio-Over-Wire

It is worth grounding this in what a photonic chip actually is at the component level, since the term gets used loosely. A photonic integrated circuit replaces copper traces with optical waveguides etched directly into the chip: a laser generates light, an optical modulator encodes data onto it, ring resonators and couplers route and filter individual wavelengths, and a photodiode converts the signal back into an electrical one at the receiving end — all stages that copper interconnects simply don’t need, and all stages that have to be manufactured, packaged, and yielded at volume for any of this to ship at AI-cluster scale. That manufacturing complexity is precisely why the foundry and packaging layer (Tower, Fabrinet, TSMC) sits at the center of this entire build-out, not on its periphery.

Photonics is the approach getting the largest share of funding and standards attention today — Nvidia, Broadcom, TSMC, and the GF Fotonix coalition are all building toward it — and it is genuinely the right tool for the longest-reach, highest-bandwidth links, where the cost and complexity of converting electrons to light is worth paying. The more interesting story for 2026, though, is what is emerging alongside it.

AttoTude — a Menlo Park-based startup, and a Wing portfolio company — has built what we think is one of the more elegant architectural answers to the copper wall: THz radio-over-wire. Rather than converting electrical signals to light and back again, its AttoEngine drives terahertz-frequency radio signals directly through ordinary wire (AttoWire), borrowing the high-frequency design discipline of photonics while skipping the optical conversion stage entirely — along with the laser sources, modulators, and photodetectors that stage requires. It is a genuinely novel piece of systems engineering: getting radio signals to behave at terahertz frequencies over wire, at the reliability and reach AttoTude reports, is a hard problem that very few teams in the world are positioned to solve.

The result is a technology with real, differentiated advantages where it counts. The company reports per-lane speeds up to 448 Gb/s and reach up to 40 meters directly over wire — covering a substantial share of intra- and inter-rack distances — with fewer conversion stages translating directly into better reliability and lower power draw per bit than an optical link of comparable reach. That is not a marginal efficiency story; removing an entire conversion stage from the signal chain is the kind of structural advantage that compounds at scale. The company has raised $91 million to date across seed, Series A, and a $50 million Series B in 2026, and has begun a signal-analysis collaboration with Keysight — concrete signs of a credible, fast-moving, well-capitalized program with a genuine technical edge, not a paper architecture chasing a trend.

The practical picture for the AI data center of 2028–2030 is a layered fabric, not a winner-take-all contest: photonics carrying the longest-reach, highest-bandwidth links where its cost is justified, and AttoTude’s THz radio-over-wire taking on the shorter- and medium-reach links — a much larger share of total connections by sheer count — where a simpler, more reliable, more power-efficient signal chain is the deciding factor. Both approaches expand the addressable interconnect market rather than splitting a fixed one. We think AttoTude’s approach is one of the more capital-efficient and structurally advantaged ways to attack that expanding market, and it is the one we have backed.

On the Horizon: MicroLEDs — the Step After NPO

The sequencing logic that makes VCSEL/NPO a credible bridge today points to a follow-on question: what comes after CPO and NPO both reach volume? The most-cited answer at OFC and Computex 2026 was microLED-based optical interconnects — and it’s worth naming for exactly that reason, not as a standalone bet. Companies including Avicena, Kopin (in partnership with Fabric.AI), and MediaTek showcased microLED architectures, pitching them as a lower-power, higher-density alternative to laser-based optics for the shortest-reach, highest-count links inside the rack — precisely the segment where Tower’s and Fabrinet’s foundry constraints bite hardest, and where a non-laser-based approach could meaningfully ease the precision-assembly bottleneck described above. Fabric.AI has targeted a late-2026 demonstration of its “Neural I/O” platform, and at OFC 2026 every hyperscaler in attendance reportedly named I/O density as their central interconnect concern — with most specifically naming microLEDs as the candidate for that next step.

The honest framing: microLEDs are real enough to take seriously as a multi-year theme, and early enough that naming a winner today would be guessing. They belong on the watch-list — not as a fourth horse in this race, but as the clearest signal that the industry already assumes today’s CPO/NPO generation is a way-station, not an endpoint. That assumption, more than any single technology bet, is what should anchor a multi-year view of this space: the interconnect problem is durable enough that the industry is already funding its own replacement before the current generation has even shipped at volume.

Lumentum and Coherent: The Incumbent Optical Suppliers Re-Rating on AI

Lumentum and Coherent are the two large-cap, pure-play optical component companies riding the photonics side of this cycle most directly, and their recent results show the re-rating happening in real time.

Lumentum’s fiscal Q3 2026 revenue rose 90% year-over-year to $808.4 million, with non-GAAP gross margin and operating margin both hitting record highs of 47.9% and 32.2% — a margin profile that looks far more like a systems company than a commodity transceiver vendor. AI and cloud infrastructure now account for over 60% of Lumentum’s revenue, and management guides to roughly 85%+ year-over-year growth, anchored by two emerging product lines: optical circuit switches (backlog exceeding $400 million) and co-packaged optics, where it has booked a multi-hundred-million-dollar CPO order for delivery in the first half of calendar 2027.

Coherent posted Q3 FY26 revenue of $1.80 billion, up 27% year-over-year, with its Datacenter & Communications segment now generating roughly 75% of total revenue — effectively transforming what was once a diversified laser and materials company into an AI-datacenter-infrastructure supplier.

Both illustrate the same underlying shift: optical components have moved from a line item inside networking budgets to a primary determinant of how many GPUs a data center can actually utilize. The companies that can manufacture at the precision and volume AI clusters require are capturing share and margin simultaneously — and that combination, growth plus margin expansion, is the signature of a re-rating, not a cycle.

It would be incomplete, though, to frame this as a US-only story. Innolight (Eoptolink’s larger rival) is the global volume leader in 800G optical modules, with roughly 35% share of 2026 shipments, and posted FY2025 net profit up 108% year-over-year (and a further 262% in Q1 2026) — figures that rival or exceed the Western incumbents’ growth rates on a much larger unit base. Together, Innolight and Eoptolink supply an estimated 60% of Nvidia’s incremental 800G module orders, which means two Chinese manufacturers — not Lumentum or Coherent — currently control the single largest swing factor in how fast Nvidia can actually ship racks. Innolight’s 800G silicon photonics modules reportedly run yields above 92% at under 14W per module; Eoptolink’s linear pluggable optics cut power roughly 50% versus conventional designs and have already won orders from Meta and Amazon. The uncomfortable fact sitting underneath the Lumentum/Coherent re-rating story, then, is this: two Chinese manufacturers — not the Western component names investors gravitate toward — currently control the single largest swing factor in how fast Nvidia can actually ship racks, and the US re-shoring efforts described above are aimed at changing that over a multi-year horizon, not next quarter.

The Manufacturing Layer: Tower and Fabrinet Hold the Real Bottleneck

Behind the component brands sits a manufacturing and foundry layer that is arguably the more interesting story about who actually controls supply — because Tower Semiconductor and Fabrinet are, functionally, the bottleneck the entire interconnect re-rating runs through. Every optical architecture discussed in this piece — Nvidia’s CPO roadmap, Lumentum’s and Coherent’s component lines, Marvell’s and Broadcom’s switch programs — ultimately needs precision photonic and analog assembly at volumes that only a small number of foundries on earth can deliver. (AttoTude’s radio-over-wire approach is a structural exception worth noting here: by skipping the optical conversion stage entirely, it sidesteps this specific bottleneck — one more way its architecture diverges from, rather than competes directly with, the photonics supply chain.) Tower and Fabrinet sit directly across that chokepoint, which is what gives their backlogs the multi-year visibility and pricing power described below.

Tower Semiconductor has signed $1.3 billion in silicon photonics contracts for 2027 delivery from its largest customers, backed by $290 million in prepayments for capacity reservation — a financing structure that signals real scarcity, not speculative demand. Tower is guiding to a 2028 model of $2.8 billion in revenue and $750 million in net profit, levered substantially to its silicon photonics and silicon-germanium BiCMOS platforms.

Fabrinet, the contract manufacturer that assembles a large share of the industry’s optical transceivers, reported Q2 FY26 optical communications revenue up 29% year-over-year to roughly $833 million, with management projecting AI optics programs alone will exceed $150 million per quarter and that its new Chonburi Building 10 will push next-generation AI optics to over 50% of total revenue exposure.

These are capacity-constrained businesses in a capacity-constrained industry — precisely the setup that produces durable pricing power and multi-year revenue visibility once a backlog is booked. And it is a layer that benefits regardless of which architecture ultimately wins more share, since photonics, VCSEL/NPO, THz-over-wire, and microLEDs all require precision packaging at volumes only a handful of foundries can deliver.

The Architecture Race: Who Controls the Next Layer Up

The next layer up — system architecture, not component supply — is where the long-term winners will actually be decided, and 2026 produced the clearest signal yet of how seriously the largest semiconductor incumbents, switching vendors, and venture-backed challengers all take it.

Marvell laid out the most explicit thesis of the year. At Computex 2026, CEO Matt Murphy argued that AI infrastructure has moved through two prior bottlenecks — first compute (which built Nvidia), then memory — and that connectivity is next, because compute performance gains are now outpacing networking gains. He positioned Marvell as the only company able to address the entire connectivity stack of an AI data center, “from millimeter scale inside the package to kilometer scale between data centers,” and pointed to roughly $36 billion in cumulative M&A — including Inphi, Cavium, and now Celestial AI — as the basis for that claim.

It’s an assertive pitch — worth weighing against the fact that a CEO laying out a “we’re the only ones who can do this” thesis at a flagship keynote has every incentive to make the addressable market sound as large, and Marvell’s position in it as singular, as possible. The technical case behind it is more concrete: Marvell detailed the engineering stack behind its Coherent optical modules — fourth-generation silicon photonics, advanced CMOS DSP, and silicon-germanium broadband analog, integrated together rather than sourced piecemeal — which is the kind of vertically-integrated stack that’s genuinely hard to replicate, and the strongest evidence for the thesis independent of how Murphy chose to frame it.

That M&A thesis runs directly through Celestial AI, which Marvell has now brought in for a headline value of roughly $5.5 billion — though a meaningful share of that figure is performance-contingent earnout rather than committed capital. The acquisition buys Marvell the “Photonic Fabric” technology, built around an “Optical Multi-Chip Interconnect Bridge” that lets any point on one die connect optically to any point on another — the clearest evidence yet that the largest players see optical chip-to-chip interconnect as core infrastructure worth owning outright, not a component to be sourced.

Marvell also used Computex to ship hardware, not just thesis: the Teralynx T100, the industry’s first 102.4 Tb/s switch purpose-built for AI data centers, runs on a 3nm process, draws under 1,000W, and delivers roughly 25% lower power than competing solutions — available in conventional pluggable, co-packaged-copper, and co-packaged-optics configurations side by side, which is itself a tell about how unsettled the near-term architecture question still is even for the companies building it.

Around Marvell’s moves, the rest of the field is moving fast too. Lightmatter raised $155 million in a Series C and has positioned its “Passage” 3D photonic interposer — featuring dense wavelength-division multiplexing and co-packaged optics — to target supercomputers scaling to roughly 300,000 nodes. Ayar Labs, backed by both Nvidia and AMD, closed a $500 million Series E in March 2026 (total funding $870 million, valuation $3.75 billion) explicitly to scale volume production of optical chiplet interconnects — a clear signal that the largest compute vendors see independent optical I/O providers as strategically necessary rather than redundant to their in-house programs. Broadcom, pursuing both CPO and the nearer-term VCSEL/NPO bridge from the switch-ASIC side, and Marvell, now doubling down through Celestial AI and the GF Fotonix platform (alongside GlobalFoundries, Cisco, and Nvidia), are — per IDTechEx — locked in what is likely the defining strategic contest in the sector through the end of the decade.

Arista Networks is making the same bet from the switching side, with a pluggable-first lens. Rather than racing straight to co-packaged optics, Arista is pushing high-density liquid-cooled pluggable optics — its new XPO platform reaches 12.8 Tbps per module, a 4x front-panel density jump over current 1.6T OSFP designs (204.8 Tbps per OCP rack unit), while management says the architecture cuts AI networking racks by up to 75% and floor space by up to 44% versus conventional pluggable deployments. Arista’s framing matters because it is effectively a referendum on sequencing, not direction: the company is betting that liquid-cooled, ultra-dense pluggables can absorb several more product cycles of bandwidth growth before the industry is forced into the harder packaging and yield problems CPO still has to solve. That’s a more conservative read of the same transition Nvidia and Broadcom are pricing in by going straight at co-packaged designs — and for investors, Arista is the cleanest test of whether “good enough, shipping now” beats “structurally better, shipping later” in this specific cycle — and the fact that a switching incumbent of its scale is still betting heavily on pluggables is itself a data point that the CPO transition will be slower and more contested than the most aggressive roadmaps imply.

One more layer worth tracking beneath the architecture race: multi-wavelength lasers, which attack the bandwidth problem by putting more colors of light on a single fiber rather than more fibers in a bundle. Xscape Photonics raised $37 million in new funding in March 2026 (bringing total Series A funding to $81 million) and launched FalconX, an eight-wavelength laser device for AI data center networks, with a roadmap (ChromX) targeting 128+ wavelengths on a single chip — developed in partnership with Tower Semiconductor, reinforcing just how central that foundry layer is to every approach in this piece, not only CPO. Enlightra, a 25-person Swiss startup, raised $15 million to build chip-scale 8- and 16-channel multiwavelength lasers explicitly aimed at replacing copper in AI chip interconnects. The logic is straightforward and compounding: every additional wavelength multiplies the data a single fiber and a single set of transceivers can carry without adding fiber count, optical connectors, or rack space — which is exactly the kind of bandwidth-per-dollar and bandwidth-per-watt lever that becomes disproportionately valuable as I/O density, not raw laser count, becomes the binding constraint inside the rack. It is early-stage venture risk today, but the structural appeal is real: scaling bandwidth by adding wavelengths to existing fiber, rather than by adding fiber count or transceiver volume, is one of the few levers that can grow capacity without a proportional increase in the precision-assembly work the rest of this supply chain depends on — which is also why Tower, sitting at that assembly chokepoint, has chosen to co-develop the technology with Xscape rather than simply wait to manufacture it.

Implications for Investors and Builders

For investors, the read is this: the interconnect layer is shifting from a historically cyclical, commoditized, margin-poor sub-sector of optical networking into a genuine control point in the AI infrastructure stack — the same dynamic that turned HBM, advanced packaging, and power infrastructure into structural winners over the last three years.

The companies best positioned sit at genuine bottlenecks across multiple physical approaches at once: precision manufacturing capacity (Tower, Fabrinet), systems-level integration with the compute roadmap (Lumentum, Coherent, Broadcom, Marvell), and novel I/O architectures co-designed with the largest buyers — spanning photonic chiplets (Ayar Labs, Lightmatter, Celestial AI/Marvell) and the complementary THz radio-over-wire layer (AttoTude) — with VCSEL/NPO and microLEDs as two further approaches still establishing where, and how durably, they fit into that picture.

Two risks deserve explicit weighting. The first is timing: much of this revenue is backlog and 2027–2029 guidance, not booked trailing revenue — Marvell’s own Celestial AI numbers don’t ramp until late 2028, and a meaningful share of even the headline deal values (as with the “$5.5 billion” Celestial AI figure) is contingent, not committed. The second is concentration: a large share of demand traces back to a small number of hyperscaler and frontier-lab capex decisions that could shift abruptly, and a downturn in any one of them would ripple through the entire stack faster than diversified end markets would absorb it.

For founders and engineers, the message is more direct: interconnect bandwidth per watt, not raw FLOPs, is becoming the design constraint that determines which architectures are buildable at scale. Teams that treat the interconnect — whether optical, radio, VCSEL-based, or some hybrid — as a systems-design primitive co-designed from day one, rather than a procurement decision made late in the build, will hold a structural advantage in both cost and achievable cluster size over the next three to five years. The fact that Nvidia, Broadcom, and Marvell are each shipping multiple interconnect formats simultaneously is itself the clearest evidence that no one — including the companies with the most at stake — believes this question is settled yet.

Sources: Tom’s Hardware, Nvidia Newsroom and Technical Blog, Broadcom (company blog on VCSEL-based Near-Package Optics), IDTechEx, Digitimes, IEEE Spectrum, SemiEngineering, OFC 2026 and Computex 2026 industry coverage (Semtech, TSP Semiconductor, EE Times, ServeTheHome, SemiconAlpha, The Register, LEDinside, Converge Digest, Insight Media), SEC filings and investor releases (Lumentum, Coherent, Tower Semiconductor, Marvell, Arista Networks), Fabrinet investor commentary (Quiver Quantitative), Ayar Labs press materials (BusinessWire, DatacenterDynamics, The Register), Celestial AI and Marvell deal coverage (optics.org, Converge Digest, Optica/OPN, Marvell investor relations), Lightmatter coverage (TSP Semiconductor), Arista XPO and OFC 2026 coverage (arista.com, COMNEN), Xscape Photonics and Enlightra funding coverage (BusinessWire, HPCwire, The Quantum Insider, SiliconANGLE), AttoTude company site and press materials (attotude.com, BusinessWire, Keysight Newsroom), hyperscaler capex coverage (Goldman Sachs, Futurum Group, Fortune), and silicon photonics market sizing from Mordor Intelligence, MarketsandMarkets, Polaris Market Research, Precedence Research, GMI Insights, and Grand View Research. Disclosure: Wing is an investor in AttoTude.

We first started writing about hyperscaler ASIC programs and the Google TPU back in early 2024. The ASIC programs have expanded massively since. This post maps every layer of the TPU supply chain: silicon design, fabrication, packaging, memory, optical interconnects, and system assembly — with the key beneficiaries at each layer.

The popular narrative is that Google does everything themselves. The reality? Their infrastructure relies on a quiet, but critical, ecosystem of hardware partners. The architecture is Google’s. The implementation is not.

These design partners feed into a five-layer supply chain extending from chip architecture through to server rack assembly across six countries.

Architecture: What Ironwood Actually Is

Before mapping the supply chain, the architecture matters. Google’s seventh-generation TPU — Ironwood (TPUv7) — represents a fundamental design break, and its choices determine who wins supply chain share.

Each chip delivers 4.6 petaFLOPS of dense FP8 — ahead of Nvidia’s B200 at 4.5 PFLOPS — with 192 GB of HBM3E at 7.4 TB/s bandwidth via eight stacks. Manufactured on TSMC N3P with CoWoS advanced packaging, Ironwood is a chiplet design with two dies connected via a 6x-speed die-to-die link. The defining architectural bet is the network: 9,216-chip superpods interconnected via a 3D Torus topology and Google’s Apollo Optical Circuit Switch (OCS) — an all-optical fabric where a single switch draws ~100W versus ~3,000W for traditional electronic switching. At pod scale, that delta is structural.

2026 TPU shipment estimates span 3.3 million units on the supply-constrained end (DigiTimes Research, citing CoWoS packaging limits) to 4.6 million on the high end (Bank of America). The range reflects a genuine CoWoS allocation constraint, not demand uncertainty — Google’s order book is not the binding variable. ASIC server shipments overall are forecast to grow 64.2% in 2026 versus 43.8% for GPU servers, with Google accounting for the dominant share among hyperscalers.

2026 projected ASIC accelerator shipments by hyperscaler. Source: DigiTimes Research, TrendForce.

Silicon Design Partners

Google owns the TPU architecture and software stack. The implementation work is distributed across a growing set of ASIC design partners — all fabbing at TSMC — with deliberate competitive tension built in.

Broadcom — primary design partner

Broadcom translates Google’s architecture into a manufacturable ASIC layout, contributes high-speed SerDes IP, handles power delivery, and coordinates CoWoS packaging with TSMC. The structural position is the story: the contract runs through 2031, Broadcom carries a $73B AI backlog scheduled for delivery over the next 18 months, and the company holds >70% market share in custom AI accelerators globally. The financial trajectory follows from that position. Full-year FY2025 AI revenue reached $20B (+65% YoY); Q1 FY2026 came in at $8.4B (+106% YoY), with Q2 guided to $10.7B. JPMorgan forecasts $50B+ for full-year FY2026 — ~150% growth. CEO Hock Tan has stated “line of sight to $100B in AI chip revenue in 2027,” with Google and Anthropic representing the dominant share.

MediaTek — inference design partner

TPUv8 splits training and inference at the silicon level. Based on supply chain research (neither Google nor MediaTek has made a public announcement), MediaTek is designing the TPU 8i (”Zebrafish”) — a single compute die plus one I/O die plus six HBM3e stacks, targeting TSMC 2nm in late 2027. The cost profile is the thesis: Zebrafish targets 20–30% lower cost than the training variant. As inference grows to 75–80% of all AI compute cycles by 2030 (per industry estimates), a purpose-built, cost-tuned inference die has durable demand. For MediaTek, it is the entry ticket into hyperscale custom silicon — a market TrendForce sizes at $118B by 2033.

Marvell — in negotiation

Google is in active discussions with Marvell for two chips: a memory processing unit and an inference-optimized TPU variant. No contract is signed as of mid-2026. Marvell’s custom silicon business runs at a $1.5B annual rate across 18 cloud-provider design wins. Adding Google would be transformative. The multi-partner strategy — Broadcom for training, MediaTek for inference, Marvell for edge inference and memory processing — creates pricing competition and eliminates single-source capacity risk.

Fabrication and Packaging

TSMC manufactures every Google TPU. Ironwood runs on N3P; TPUv8 targets N2, with tape-outs planned for 2026–2027. TSMC also handles CoWoS advanced packaging — the step integrating compute chiplets with HBM3e stacks. TSMC is targeting 140,000 CoWoS wafers per month by end-2026, a 4x expansion from early 2024. Even at that scale, CoWoS remains an allocation market — which is why supply chain researchers and Bank of America disagree on 2026 TPU volumes by more than a million units.

CoWoS packaging requires ultra-high-density ABF substrates. Ibiden and Shinko Electric dominate this layer. AI server substrates carry approximately 3x the pricing premium of standard server substrates due to layer count and routing complexity. New fab construction takes 2–3 years and hundreds of millions in capital — supply cannot respond quickly to demand.

High-Bandwidth Memory

Ironwood’s 192 GB per chip, 8-stack HBM3E configuration makes HBM arguably the most consequential component constraint in the program. The chiplet design — two compute dies connected via a die-to-die interface rather than one monolithic die — is partly dictated by TSMC’s reticle size limit at N3P, but it also maximizes the number of HBM stacks each die can directly service: four stacks per die, each connected via a dedicated 512-bit memory bus. That physical arrangement is why HBM supply, not compute logic, is the effective ceiling on Ironwood production volumes.

Samsung holds approximately 60% of HBM3E supply to Google via Broadcom for Ironwood, with HBM4 already qualifying through Broadcom’s test process for the next generation. SK Hynix supplies the remaining ~40%. Micron is not a meaningful factor in this supply chain as of mid-2026.

Optical Interconnects — a differentiated layer

This is the layer most investors underestimate. Google’s Apollo OCS architecture is a full architectural break from conventional switching, and the qualified supplier base is narrow.

Lumentum — Apollo OCS system

Lumentum supplies the MEMS-based optical circuit switches at the heart of Apollo. These switches use micro-mirrors to enable direct fiber-to-fiber connections, eliminating optical-electrical-optical conversions. A single Apollo switch draws ~100W versus ~3,000W for a comparable electronic switch — a 95% power reduction. Lumentum’s primary commercial product is a 300×300 OCS at approximately $150,000 per unit (~$500/port). In 2025, Google deployed an estimated 12,000–15,000 units. Lumentum is the sole-source supplier for this component; there is no qualified alternative in production.

Innolight and Eoptolink — 800G+ transceivers

Every cross-cabinet link in an Ironwood pod runs on 800G or 1.6T optical transceivers. TrendForce projects Innolight — the larger of the two and Google’s primary silicon photonics collaborator on 1.6T development — will capture ~80% of Google’s 800G+ optical module orders in 2026. Eoptolink captures most of the remainder; it is a second-tier supplier by volume but strategically important for supply chain resilience. The company posted 179% revenue growth in 2024, driven by cloud operator direct sales, and has built a manufacturing facility in Thailand specifically to serve North American demand outside Chinese export-control exposure. Collectively, Google’s Ironwood architecture is projected to push 800G+ transceiver share above 60% of all optical module demand by 2026 (TrendForce).

System Assembly — Server ODMs

TPU chips and optical components must be integrated into server platforms and racks. Taiwan-based ODMs dominate this work. Inventec is the most clearly confirmed primary ODM for Google’s TPU server builds, with management publicly expressing confidence in double-digit ASIC server shipment growth in 2026. Wiwynn (a Wistron subsidiary) is actively expanding SMT production capacity in Malaysia for ASIC server assembly. Quanta, Foxconn, and Mitac are all competing for additional allocation.

ASIC server shipments are forecast to grow 64.2% in 2026 versus 43.8% for GPU servers. The margin dynamic favors ODMs: ASIC servers carry lower unit prices than GPU platforms but require more customized design work and longer production runs, which compresses per-unit revenue while improving margin mix. The net effect is that ODM revenue from ASIC servers grows more slowly than volume, but profitability improves as ASIC share of the revenue mix increases.

Beneficiary Summary

Closing Thesis

The companies with clearest structural exposure are Broadcom (silicon implementation, locked through 2031), Samsung (HBM memory, primary qualified supplier), and Lumentum (optical switching, sole-source). These are not cyclical trades on AI capex — they are design partners with multi-year roadmaps and switching costs measured in years.

At $175–185B in committed 2026 capex and 3.3–4.6M projected TPU shipments, the question is no longer whether Google’s custom silicon program is real. The question worth stress-testing is what breaks the thesis. Three scenarios are worth watching: first, CoWoS yield issues at TSMC N2 that compress 2027 volumes below the 5M unit consensus — this directly impairs every vendor in the stack. Second, Samsung HBM4 qualification delays that concentrate more supply at SK Hynix and introduce a single-source risk Google has spent years trying to avoid. Third, Nvidia closing the inference cost-per-token gap faster than expected through Blackwell derivatives — this doesn’t threaten Broadcom’s training revenue, but it limits how aggressively Google can price TPU inference access externally, which is increasingly the commercial justification for the capex.

None of these scenarios break the supply chain. They delay it and compress near-term returns for the vendors most dependent on volume ramp. The structural position of the Tier 1 beneficiaries — design lock-in, sole-source components, multi-year agreements — survives all three. What changes is the timeline, not the destination.

Sources / Recommended readings:
SemiAnalysis: Google TPUv7 — The 900lb Gorilla
Google Cloud: Inside the Ironwood TPU codesigned AI stack
The Next Web: Google assembles four-partner chip supply chain
TrendForce: Global AI Server Shipments Forecast 2026
TrendForce: Google's 800G+ Optical Transceiver Push
KED Global: Samsung leads HBM for Google Ironwood (~60% share)
DigiTimes: Google TPU stays strong, Inventec confident 2026
DigiTimes: Google TPU demand vs CoWoS and memory constraints
Gazettabyte: Lumentum OCS for AI data centres
Oplexa: Broadcom Google TPU Deal 2026
CNBC: Alphabet plans $80B equity raise
Broadcom: FY2025 Annual Results
Tom's Hardware: Custom AI ASICs — state of play, May 2026

TL;DR — Five things to know

1. AI racks go from 120 kW today to 600 kW by 2027. Power infrastructure value per rack jumps from $36K to $398K+ — more than 10× baseline. This is not incremental.

2. The architecture shifts completely — AC→50V DC becomes AC→800V DC (HVDC). Silicon Carbide and GaN are the enabling materials. NVIDIA has already picked its semiconductor partners.

3. Air cooling is effectively dead above 40 kW. The liquid cooling market grows from $5.5B to $25.8B by 2035. Every new hyperscale campus is being designed liquid-native from the ground up.

4. Power transformers now carry 4-year lead times. More than 50% of U.S. data centers planned for 2026 face delays. This is a structural bottleneck, not a supply chain hiccup.

5. The investment thesis spans five layers: grid infrastructure → facility power → rack PSU → power semiconductors → chip-level VRM. Each has different risk, margin, and timing.

Five Generations. Five Times the Power. Twenty-Four Months.

A GB200 Blackwell rack today draws 120 kilowatts. That’s already forcing hyperscale operators to redesign power distribution floors and pull in supplemental cooling hardware mid-build. GB300 takes that to 140 kW. Vera Rubin, shipping in 2026, clears 200 kW. Vera Rubin CPX hits 380 kW. And Vera Rubin Ultra — paired with NVIDIA’s new Kyber rack architecture — lands at 600 kW per rack.

Six hundred kilowatts. In a single rack. Roughly the power draw of 600 American homes, packed into a steel enclosure seven feet tall and two feet wide. A 1,000-rack AI cluster at Vera Rubin Ultra density requires 600 megawatts of utility-grade power — the average electrical load of San Francisco.

The economic consequence, per Morgan Stanley Research, is equally direct. Power infrastructure value per AI server rack jumps from $36,000 with GB200 to $398,160 with Vera Rubin CPX — a 1,005% increase — and exceeds 10× baseline with Vera Rubin Ultra. The power electronics industry isn’t a side story in the AI buildout. It’s one of the central economic narratives.

Source: Morgan Stanley Research, Exhibit 7

The Architecture Has to Change. Completely.

Today’s data centers run on a power architecture that was never designed for what’s being asked of it. AC from the grid converts to low-voltage DC inside each server’s PSU. Fine at 10 kW per rack — a number that grew only incrementally for decades. Now it’s growing exponentially, and at 600 kW the physics turn against you hard.

Source: AEIS

The problem is Ohm’s law. Power loss in a conductor equals current squared times resistance (P = I²R). Low-voltage DC forces you to move the same wattage at much higher current, and losses scale with the square of that current. The copper in your busways heats up. The conversion stages multiply. A system losing 8–10% of its power at 120 kW is wasting 48–60 kW per rack at 600 kW — waste heat equivalent to the HVAC load of a small office building, generated inside a 7-foot cabinet.

NVIDIA’s answer, baked into Vera Rubin CPX and Ultra, is a complete switch to High-Voltage Direct Current at 800V. A centralized Silicon Carbide rectifier converts utility AC directly to 800V DC. High-voltage distribution across the rack cuts resistive losses sharply. Fewer conversion stages. Less heat. More compute per watt delivered.

This is a platform transition, not an upgrade cycle — comparable in kind to spinning disk to NAND, or mechanical relays to solid-state switching. Every piece of rack-level power infrastructure gets redesigned. Every PSU, busbar, connector, and voltage regulator module needs re-qualification for 800V operation. That qualification cycle runs 12–24 months per generation. Vendors who clear it first hold a durable position in that GPU lifecycle. Vendors who don’t are excluded from the design entirely.

Vertiv has a full 800V HVDC product line slated for 2026. ON Semiconductor and Navitas Semiconductor have both disclosed formal NVIDIA collaboration agreements for the 800V transition — the kind of design-in that locks a vendor into a GPU generation’s revenue cycle for 3–5 years.

Silicon Carbide and GaN: The Materials That Make It Possible

Silicon has a hard ceiling around 650V. It cannot efficiently switch at the voltages and speeds 800V HVDC demands. The answer is wide bandgap semiconductors — materials with a larger energy gap between valence and conduction bands, which translates directly to higher breakdown voltage, higher switching frequency, and lower on-resistance.

Two materials carry the load. Silicon Carbide handles the high-voltage, high-current front end of HVDC conversion. A 1200V SiC MOSFET replacing a legacy silicon device delivers a 25–40% reduction in conversion losses at the rectification stage. Gallium Nitride runs the other side — operating at extremely high switching frequencies in the DC-DC stages that deliver final voltage to the GPU die, enabling smaller magnetics and better transient response in a fraction of the footprint.

NVIDIA’s vendor selections are the clearest market signal available. Navitas Semiconductor’s GaNFast and GeneSiC technologies are in the reference design for Vera Rubin generation power systems. ON Semiconductor’s SiC MOSFET collaboration targets the 800V rectification stage specifically. Infineon — the world’s largest power semiconductor company at roughly €16 billion in annual revenue — has 1200V SiC devices in active qualification for AI data center applications, with manufacturing scale no smaller peer can replicate at hyperscale volumes.

The substrate bottleneck. Most SiC device manufacturers — ON Semi, STMicro, Infineon — depend on SiC wafer supply from a tight group of substrate producers. Wolfspeed was the dominant U.S. source. That thesis now requires careful handling: Wolfspeed filed for Chapter 11 in early 2025, crushed by its aggressive capacity expansion program colliding with softening EV demand. The substrate leverage position is still structurally real — the wafers are still needed, the manufacturing assets didn’t vanish — but the equity is a restructuring story, not a clean infrastructure compounder. Meanwhile, Chinese manufacturers (SICC, TankeBlue, Sanan IC) are expanding capacity aggressively, applying downward pressure on substrate pricing that will ultimately benefit device makers and end customers while compressing whoever inherits Wolfspeed’s supply position post-reorganization.

Air Cooling Is Already Dead. The Industry Just Hasn’t Filed the Death Certificate.

Conventional air cooling has a practical density ceiling around 30–40 kW per rack. Above that, the volume of air you’d need to move exceeds what you can force through a rack enclosure at reasonable static pressure. Fans consume more power. Hot spots form. The economics invert — you spend more electricity removing heat than you would have spent throttling the compute.

At 120 kW, operators are already deploying rear-door heat exchangers and direct-to-chip liquid loops as a patch. Vera Rubin at 200 kW makes direct-to-chip mandatory. CPX at 380 kW pushes into immersion cooling territory. Vera Rubin Ultra at 600 kW strongly favors two-phase immersion — liquid refrigerant that boils on contact with hot components, with vapor recondensed and recirculated — though high-flow single-phase direct liquid cooling remains physically viable depending on the specific thermal design. This is industrial cooling technology borrowed from semiconductor fabs and chemical plants, now being deployed at internet infrastructure scale.

The capital intensity here is severe, and not in a “spend more to upgrade” sense — in a “start over” sense. A facility built for 40 kW air-cooled racks cannot be incrementally retrofitted to handle 200+ kW liquid-cooled systems. Floor loading, piping, leak detection, CDU placement, and electrical distribution all have to be redesigned from scratch. Greenfield hyperscale campuses announced today are going in liquid-native. Legacy colo providers face a binary: spend heavily to rebuild, or cede the AI workload tier entirely to the hyperscalers who built for this from day one.

The market trajectory reflects this. Liquid cooling at roughly $5.5 billion in 2025 is projected to reach $25.8 billion by 2035 — approximately 18–20% CAGR, driven almost entirely by AI infrastructure density requirements. Cloud providers dominate the current installed base, but enterprise AI deployments will extend that demand well beyond the hyperscale tier.

Transformers: The Four-Year Bottleneck Nobody Talks About Enough

Every kilowatt of AI compute connects to the utility grid through a transformer. Not metaphorically. A physical transformer — a custom-engineered device wound with high-purity copper and grain-oriented electrical steel, filled with insulating oil, weighing anywhere from 5 to 500 tons — sits at the boundary of every data center and the grid that feeds it. There is no shortcut around this device.

High-capacity units now quote 4-year lead times. Standard power transformers are at 128 weeks. Generator step-up units (GSUs) are at 144 weeks. Prices for power transformers have risen 77% since 2019; GSUs are up 45%. GSU demand itself increased 274% between 2019 and 2025. More than 50% of U.S. data centers planned for 2026 are expected to be delayed or canceled specifically because specialized electrical equipment cannot be sourced in time.

The causes are structural. U.S. transformer manufacturing capacity hasn’t meaningfully expanded in 15 years. The critical raw material inputs — grain-oriented electrical steel (GOES) and high-purity copper — are sourced primarily from Japan, South Korea, and increasingly China. Building a new transformer factory takes 3–5 years from capital commitment to first unit out the door. Nobody is going to solve this in a budget cycle.

Substations compound the problem. A large AI campus needing 100–500 MW of utility interconnect requires switchgear, protection relays, metering equipment, and civil infrastructure — each with its own lead time stack. U.S. grid interconnect permitting averages 3–5 years. The interconnect queue at regional transmission organizations currently holds over 2,600 GW of pending capacity requests. Total U.S. installed generation capacity is approximately 1,200 GW. The queue is more than double the grid.

The AI Power Stack: Five Layers from Grid to GPU

Power doesn’t teleport from the utility grid to an NVIDIA GPU. It moves through five distinct layers of infrastructure, each with its own supply chain, lead times, and companies controlling the chokepoints.

Layer 1 — Grid & Substation Infrastructure High-voltage transformers, switchgear, substations. The bottleneck before the building even exists. Lead times now 2–4 years. More than 50% of U.S. data centers planned for 2026 face delays here.

Layer 2 — Data Center Power Distribution & UPS PDUs, UPS systems, busways, coolant distribution. Everything between the utility interconnect and the server rack. This is where facility-level critical power lives.

Layer 3 — Rack-Level Power & PSU Server power supply units and power shelves. Currently converting AC to 50V DC per server. The 800V HVDC transition — arriving with Vera Rubin CPX in 2027 — redesigns this entire layer from scratch.

Layer 4 — Power Semiconductors (SiC & GaN) The materials that make the 800V transition physically possible. Silicon Carbide handles high-voltage rectification; Gallium Nitride handles high-frequency low-voltage conversion. Together they deliver 25–40% fewer conversion losses versus legacy silicon.

Layer 5 — Point-of-Load & Chip-Level VRM Voltage regulators and power management ICs that deliver precise, tightly regulated voltage directly to the GPU die. Highest dollar content per AI server. Smallest physical footprint. Closest to the compute.

Each layer has different lead times, different competitive dynamics, and different exposure to the 800V transition. The companies controlling the chokepoints at each level are covered below — but the macro argument stands on its own.

The Physics Don’t Negotiate

Everything else in AI infrastructure is debatable: which model architecture wins long-term, which cloud provider takes share, which REIT lands the best interconnect deal. The power problem is not debatable. It is a direct consequence of semiconductor physics and thermodynamics — more compute per chip requires more power per chip, more chips per rack requires more power per rack, and neither of those relationships bends for anyone’s financial model.

The scale of the requirement — 600 kW per rack, 4-year transformer lead times, an HVDC transition touching every electrical component from the substation to the GPU — is measurable, predictable, and already locked into NVIDIA’s product roadmap. These are not projections. They are engineering specifications.

The companies in this stack are not speculating on AI demand. They are selling into a confirmed equipment cycle with 2–4 year backlog visibility and architectural transitions that reset design-win competition every 18–24 months. For investors willing to work through the layers, the power electronics theme may be one of the most grounded, data-supported ways to own the AI infrastructure buildout — without paying for hope.

Note: If you are a founder/operator building in this space, please reach out to me.

Three companies — SK Hynix, Samsung, and Micron — own that bottleneck. They control approximately 96% of DRAM and effectively 100% of High Bandwidth Memory (HBM). HBM is now the single most economically critical component inside Nvidia’s accelerators, contributing >50% of the bill of materials of a Blackwell Ultra (B300) and rising further on Rubin. Historically a cyclical industry, memory suppliers are seeing above prior peak-cycle gross margins today in excess of 70%.

At peak-cycle HBM economics, SK Hynix is beginning to outearn the hyperscaler businesses like Meta that buy their products. Combined, the memory triopoly is projected to generate 2–3x Meta’s net income in 2026.

Compute Scales Faster Than Memory

Every Nvidia generation has run into the same wall: memory. Memory capacity per GPU grew ~18x from P100 to Rubin Ultra in 11 years. FP compute grew approximately 500x over the same period.

The implication is structural: every successive AI generation becomes more bandwidth-constrained. Memory has gone from ~20% of GPU BOM (A100) to ~55% (B200). For Rubin, HBM may exceed the cost of the logic die itself. We are beginning to approach physics limits with how much HBM can be placed on the processor. GPU shoreline is fixed by die perimeter, and HBM stacks can only sit along two of the four edges — which is why Rubin Ultra moves to 16 stacks by going taller, not wider.

HBM Bill of Materials and Margin Capture

A B200 has 8 HBM3E stacks at 24 GB each (192 GB total). HBM share of GPU BOM is ~52% on B200, ~58% on B300 (Blackwell Ultra, 288 GB HBM3E 12-Hi), and tracking ~62% on Rubin (HBM4) based on early ASP guides.

Nvidia sells B200 at $30,000–40,000 list. Of every incremental B200 dollar, roughly $0.10–0.15 flows to HBM suppliers as gross margin contribution. SK Hynix captures ~45–50% of that. Samsung ~30%. Micron ~22%.

Memory suppliers are now tied to Nvidia’s revenue trajectory more tightly than perhaps any supplier-customer relationship in modern semiconductor history.

Cost per Gigabyte: The Brutal Hierarchy

Memory is not one market. It is a stratified stack with six orders of magnitude in cost difference.

HBM3E is ~6x DDR5 per gigabyte but ~25x the bandwidth per stack. On a $/(GB·bandwidth) basis, HBM is the only economically rational choice for AI workloads above a certain compute density.

SRAM is two to three orders of magnitude more expensive than HBM per gigabyte. And SRAM scaling has stalled. From TSMC N7 to N5, SRAM density improved 5%. From N5 to N3, less than 5% again. The historical assumption that on-die cache would keep up with logic has failed.

This is the SRAM scaling cliff. Every AI architecture that depends on large on-die caches (Cerebras, Groq-style approaches) faces an increasingly hostile cost curve. HBM remains the only memory technology that combines large capacity, high bandwidth, and acceptable cost.

The Memory Wall, Quantified

AI’s core architectural problem is no longer compute scarcity. It is memory starvation.

AI workloads are bound by three physical constraints, not by FLOPS:

1. Modern GPUs require far more compute per byte than transformer workloads naturally generate. Every GPU has a balance point — how many math operations it can do for each byte of memory it can fetch. For V100 (2017) the balance point was ~125 FLOPS per byte. For B200 today it is ~560 FLOPS per byte. To run the GPU flat-out, software needs to do that many operations on each byte it touches. Transformer decode workloads fall dramatically short of that threshold. Typical decode workloads only do 1–10 operations per byte fetched — they spend most of their time waiting on memory. The result: AI silicon often runs at 15–35% of peak FLOPS during inference. Every additional TB/s of HBM bandwidth converts directly into utilization recovery and lower cost-per-token.

2. KV cache now dominates inference memory. During inference, every prior token’s “key” and “value” representations have to stay in memory for the next token to attend to them. This is the KV cache, and it grows with context length × batch size. Llama 3 70B at 128K context with batch size 32 needs ~80 GB just for KV cache — more than the model weights themselves. At 1M context, KV cache exceeds 600 GB per request. This is why H200’s 141 GB and B200’s 192 GB are a step change versus H100’s 80 GB: not because the weights got bigger, but because the cache did.

3. Moving data costs more energy than doing math on it. Moving one bit across HBM costs about 7 picojoules. Across off-package DDR5: 80 pJ. From on-die SRAM: 0.1 pJ. Doing an 8-bit multiply on a 4 nm node: 0.05 pJ. At trillion-operation inference scales, data-movement energy exceeds compute energy by 3–10x.

   This is why every architectural response — HBM4 with logic base dies, stacked SRAM, processing-in-memory — is fundamentally about reducing joules-per-byte moved.

The economic translation: HBM bandwidth is the highest-leverage capex decision in AI infrastructure. For modern transformer workloads, the marginal value of additional HBM bandwidth often exceeds its marginal cost.

What an HBM Stack Actually Is

Source: High-Bandwidth Memory, Semi Engineering

Before getting into vendor differences, a quick engineering picture. HBM isn’t a chip — it’s a small skyscraper of chips.

An HBM3E 12-Hi stack is twelve DRAM dies sitting on top of a logic base die. Each DRAM die is ground down to about 30 μm thick — roughly a third of a human hair — so the whole tower fits inside the package. The dies are connected vertically by through-silicon vias (TSVs): copper-filled holes drilled through each die, 5–10 μm wide and spaced ~50 μm apart. A 12-Hi stack has 1,200–1,800 TSVs per layer for data and power, plus thousands more for signal integrity.

The stack sits next to the GPU compute die on a silicon interposer. The interposer is the wiring layer that carries the 1,024-bit (HBM3) or 2,048-bit (HBM4) bus between memory and compute at trace densities and impedances that organic substrates can’t match. For Nvidia accelerators, the relevant implementation is TSMC’s CoWoS-L.

Three things make HBM hard to build:

• Heat. Each stack dissipates 4–8 watts. That heat has to escape upward through twelve thin silicon layers and the adhesive between them. The thermal limit is why H100 launched in 2022 with only five of six possible HBM3 stacks active — the sixth bay was reserved as thermal and yield margin.

• Yield compounds. A single bad TSV kills the entire stack. At 99% per-die yield, a 12-Hi stack yields 88.6%. At 95% per-die, only 54%. This is why per-die yield matters more than headline density at HBM scale — and why SK Hynix’s 75–80% 1bnm yield is a wider competitive moat than the number alone suggests.

• Wider IO beats faster IO. HBM4 doubles the interface from 1,024 to 2,048 bits but keeps the per-pin speed roughly the same as HBM3 (~6.4 Gbps). Wider-and-slower uses less power per bit and yields better than narrower-and-faster at modern interposer densities. This single choice forces the base die onto a logic foundry node (TSMC N5/N3) instead of a DRAM line and determines which vendors can build it.

There are two ways to physically stack the dies:

• MR-MUF (Mass Reflow with Molded Underfill) — SK Hynix’s proprietary method. Bond and fill the gaps between dies in a single thermal cycle. Better heat conduction, fewer voids between layers.

• NCF (Non-Conductive Film) — Samsung’s method. A pre-applied film is bonded layer-by-layer. Cheaper equipment, slower per stack, lower yield historically. Samsung’s revised “Advanced TC-NCF” closed but did not eliminate the gap for HBM4.

Layer count keeps climbing. HBM4 supports 16-Hi configurations — double what HBM3 launched at. The earlier 720 μm packaging ceiling could not accommodate 16-Hi stacks, forcing the standard to expand to 775 μm in late 2024. The taller envelope means microbumps remain viable for HBM4, and the long-anticipated transition to direct copper-to-copper bonding has slipped to HBM4E or later.

Once copper-to-copper bonding arrives, interconnect pitch shrinks dramatically while removing the bump gap between dies entirely — enabling higher stack density and lower power per bit. The tradeoff is a new bottleneck: hybrid bonding tool capacity.

HBM market share, 2024–2026E. The SK Hynix monopoly era is over; Samsung's HBM3E 12-Hi qualification and Micron's ramp have compressed the leader's share by 7 points. Total HBM revenue still compounded >100% in 2025.

The Triopoly in Numbers

DRAM is a triopoly. HBM is no longer a SK Hynix monopoly — Samsung’s HBM3E 12-Hi qualification at Nvidia in Q4 2025 and Micron’s ramp have compressed the leader’s share by ~7 points in 18 months. But total HBM revenue continues to compound at >70% YoY, so all three are growing absolutely.

Through five DRAM cycles since 2010, the industry consolidated from 10+ vendors to three. Each downturn pushed marginal capacity out. The result is the highest sustained pricing power in the history of memory: HBM3E ASPs are 6–8x conventional DDR5 per gigabyte, HBM4 ASPs are ~10x, and capacity remains fully sold out through 2027.

SK Hynix — The HBM Leader

The #1 HBM supplier at ~45% share, sole supplier of HBM3 to Nvidia’s H100, and now the lead supplier of HBM4 into Rubin.

The technical moat is two-fold. SK Hynix’s 1bnm DRAM yield runs 75–80%, versus Samsung’s 60–65% on 1cnm — a gap that compounds across 12 stacked dies. Its proprietary MR-MUF stacking process keeps it alone above 80% yield on HBM3E 12-Hi.

Looking forward: SK Hynix shipped first HBM4 12-Hi samples to Nvidia in March 2026 — a 6–9 month lead over the competition into the Rubin socket. The HBM4 base die is fabricated at TSMC N5, coupling the memory and logic moats at the most advanced node.

Main risk: Nvidia is ~42% of total revenue, the tightest single-customer relationship in modern semiconductors.

Samsung — The Vertically Integrated Comeback

The largest memory company by capex, but it lost the HBM3 cycle. Thermal failures at Nvidia in 2023, yield misses at the 1anm node, and slower TC-NCF stacking left Samsung well behind.

The story flipped in 2026. 1cnm DRAM yields converged to ~65% in Q1 2026. HBM3E 12-Hi qualified at Nvidia in November 2025. “Advanced TC-NCF” closed the stacking gap. And AMD’s MI400 is now committed primarily to Samsung HBM4 — the first major AI socket Samsung has fully won this cycle.

The structural advantage no one else has: Samsung is the only memory company that also owns its own logic foundry. As HBM4E moves to customer-specific logic base dies in 2027, Samsung can co-design memory and base die without crossing a corporate boundary.

Main risk: Still 6+ months behind SK Hynix on HBM4. The HBM4E generation in 2027 is where Samsung’s integration thesis actually has to prove out.

Micron — The American Hedge

The smallest of the three, but with two distinct advantages.

A different DRAM cell architecture. Micron uses an “open bit-line” topology with vertical capacitors where SK Hynix and Samsung use planar designs. Micron claims ~30% lower active power on HBM3E at the same bandwidth (vendor-published, directionally credible).

US production. Micron is the only material DRAM manufacturer with US fabs. Idaho ID-1 opened October 2025; the New York Clay megafab was pulled forward to 2027. A $6.1B CHIPS Act grant and $7.5B in loans underwrite a multi-decade capex floor that no Asian competitor has.

Looking forward: HBM3E 12-Hi shipping into B300 since Q4 2025. HBM4 sampling at Nvidia in Q2 2026, volume H1 2027 — roughly parity with Samsung, ~6 months behind SK Hynix. HBM has gone from <2% of Micron revenue two years ago to ~26% today — a faster mix transition than either competitor went through.

Why Not More Players?

Greenfield DRAM fab cost in 2024: $15–25B. Lead time: 4–5 years. Yield ramp: 18–36 months. HBM-specific packaging capex (TSV, CoWoS-equivalent): another $2–5B per node.

Total cumulative capex required to enter the HBM market at material scale (10% share): ~$30–50B. Time required: 7–10 years.

No new entrant is plausible. Chinese efforts (CXMT, YMTC) are 3–5 nodes behind and structurally blocked from EUV. Indian, Japanese, and Taiwanese efforts are early-stage. The triopoly is durable for at least a decade.

Structural Observations

Not investment recommendations. Five things to track across the cycle:

1. SK Hynix’s HBM4 lead is the highest-margin product in memory history. Watch Nvidia concentration (42% of revenue) and hyperscaler dual-sourcing — visible already on AMD MI400 (Samsung) and Trainium 3 (Micron + Samsung).

2. Samsung’s recovery is data-supported, not speculative. 1cnm yield convergence, HBM3E 12-Hi qualification, and the MI400 win ended the “structurally behind” thesis. Open question: does foundry-plus-memory integration matter at HBM4E?

3. Micron has crossed from optionality to fundamentals. Fully booked HVM through 2026, 24% share, CHIPS Act margin floor. Main risk is HBM4 schedule slip.

4. TSMC is the silent fourth participant. CoWoS-L sold out through 2027 despite ~3x capacity expansion. Memory and logic moats are now coupled at the most advanced node.

5. Equipment vendors capture proportional margin. HBM capex intensity per wafer is 3–5x prior DRAM cycles — flowing to lithography, TSV etch, deposition, and hybrid-bonding tools.

Where Startups Are Trying to Break In

A wave of startups is either trying to route around HBM or build new abstractions on top of it. Three categories matter.

1. Memory Processing Units (MPUs). Compute placed inside the memory array itself, so data never has to cross the package. The architectural bet: as data movement comes to dominate energy and latency, the chip’s center of gravity shifts toward memory with compute as the satellite. Several early-stage startups have tested working silicon on this concept, but the category remains pre-commercial — useful as a forward indicator of where the industry may head, not yet a commercial threat to HBM.

2. Custom HBM-plus-SRAM accelerators. Transformer-specific ASICs that combine HBM for capacity with oversized on-die SRAM for bandwidth. Cerebras WSE-3 carries 44 GB of on-die SRAM at 21 PB/s aggregate — no HBM at all. Groq’s LPU goes further with ~230 MB SRAM per chip and statically schedules weights across hundreds of chips. MatX pairs HBM3E/HBM4 with large on-die cache, betting that workload specialization can beat Nvidia’s general-purpose silicon on tokens-per-dollar.

3. Memory fabrics. Connective tissue around memory rather than memory itself. Astera Labs (CXL switches and memory expanders, now public) leads the commercial wave. Eliyan’s NuLink offers organic-substrate chip-to-chip interconnect competitive with silicon interposers — potentially loosening TSMC’s CoWoS chokepoint. Enfabrica is building a converged network-and-memory fabric that pools HBM and DDR across a rack. Celestial AI is pushing photonic interconnects with claims of 10–100x lower energy per bit between memory and compute.

None of these displaces HBM as the bottleneck. Each either accepts HBM as the foundation and extracts more performance around it, or routes around HBM in workloads narrow enough for SRAM to substitute. That is exactly why the triopoly remains durable.

Summary

The memory triopoly is the most concentrated, most capital-intensive, most expensive chokepoint in the AI server. HBM is now 50%+ of accelerator BOM. Memory bandwidth grows ~2x per Nvidia generation while compute grows 5–10x. The gap widens. The triopoly captures the margin.

Nearly every architectural response — HBM4 logic base dies, stacked SRAM, PIM, MPUs, CXL fabrics — ultimately reinforces the same bottleneck. The next decade of AI capex driven by Nvidia and TSMC flows into the memory triopoly more than into any other layer of the stack.

The bottleneck is the moat.

That isn’t a 2x or 3x problem — it is an order-of-magnitude shift in machine-hours per shipped unit. Multiply across the millions of accelerators shipping into datacenters over 2025–2027 and you get a structural capacity squeeze in semiconductor test, with two companies — Advantest and Teradyne — sitting on the chokepoint.

At a glance

• Test time per die: ~50s mobile SoC → 20+ minutes for AI accelerator

• CY25 ATE TAM: $9.0B, +50% YoY

• Advantest SoC tester share: 56% → 66% in twelve months

• Teradyne Q1’26 revenue: +87% YoY, ~70% AI-driven

• Combined duopoly share: ~90%+

Why test time exploded

Four physics-driven dynamics are simultaneously inflating test surface area, test pattern count, and required test conditions. Each adds machine time linearly. Together they compound.

1. Die size at the reticle limit. Blackwell B200 packages two ~814mm² dies — each at the EUV reticle limit — connected via a 10TB/s NV-HBI interconnect. Test pattern count scales with transistor count (B200: ~208B transistors) and IO complexity. A reticle-limited die requires 2–3× the test patterns of a 7nm-era SoC.

2. Chiplet integration. UCIe and proprietary D2D fabrics let designers compose accelerators from 4–8 chiplets. Every chiplet must be screened as a Known-Good-Die (KGD) before assembly because a single defective die scraps a $50K+ package. Each D2D link adds a high-speed interconnect test pattern, and per-die screening multiplies machine hours.

3. HBM stacking. HBM3E ships as 12-Hi stacks with a 1,024-bit interface and ~1,024 TSV connections per layer. Every DRAM die is tested individually, then the full stack is validated as a Known-Good-Stack-Die (KGSD). HBM3E test time runs roughly 2× HBM2E. HBM4 (16-Hi, 2,048-bit) lengthens it again.

4. Power and thermal. Top-line AI accelerators draw 700–1,200W. Test sockets must dissipate that thermal load while maintaining clean signal integrity, and the chip must be exercised under representative AI workloads — not just static patterns — to surface latent defects. This forces coolant-integrated handlers and longer at-load test intervals.

The aggregate effect: cost of test as a percentage of chip ASP has roughly tripled on leading-edge AI parts — from historical levels of <2% toward 5–7% on the most complex packages. The customer can’t do anything about it. Yield economics on a $30–70K chip mean any defect that escapes test costs orders of magnitude more than the test itself. The bill scales linearly with units shipped, and it accrues almost entirely to the two companies whose platforms are qualified for the job.

A duopoly with structural lock-in

Advantest’s V93000 is the qualified SoC tester at Nvidia, AMD, and the merchant ASIC supply chain (Broadcom-Google, Marvell-Meta/Microsoft). Teradyne’s UltraFLEX dominates Apple silicon and the broader mobile/auto base.

The lock-in comes from the test program itself — a software artifact representing 6–18 months of engineering work, deeply integrated with the chip’s design flow, that gets re-used across every wafer for the chip’s full lifecycle. Once a fab and OSAT qualify a tester, replacement risk approaches zero.

That’s why the recent share dynamics matter so much: Advantest gained 10 percentage points of SoC tester share in CY25 alone — from 56% to 66% — by sitting inside the AI accelerator supply chain at a moment when AI accelerator volume is the only segment of semiconductors growing meaningfully. Total tester market share moved from 58% to 65%. Teradyne gave up incremental SoC share but is monetizing the same demand wave through its broader portfolio: system-level test, photonics, robotics-assisted manufacturing.

The financial signature

The technical bottleneck shows up cleanly in the sales and margin figures.

Advantest FY25 (year ended March 2026): operating margin moved from 29.3% to 44.2% — a 1,490bp expansion in twelve months on revenue of $7.5B, +47% YoY in USD terms. Operating income roughly doubled to ~$3.3B. Q4 ran at 46.7% operating margin on 67.4% gross.

Teradyne Q1 2026, annualized: the run-rate is $5.1B in revenue, ~$1.9B in operating income, and $10.24 in non-GAAP EPS — versus a $2.7B revenue / ~$0.6B op-income / $3.00 EPS run-rate one year earlier. Non-GAAP operating margin moved from 20.5% to 37.5%, a 1,700bp YoY expansion.

Both companies are now operating inside or above their long-term target margin bands. This isn’t volume leverage alone — it is pricing on complex testers in short supply, the financial fingerprint of a duopoly with a captive AI accelerator customer base.

Where the next leg of growth comes from

The interesting part of the story isn’t ATE itself. It’s the adjacent test layers that have emerged because traditional ATE no longer catches every defect that matters in an AI workload.

• System-Level Test (SLT). Exercises the chip in conditions close to its end-use — running real workloads, at full thermal load, with real power delivery — to catch latent defects that ATE patterns miss. SLT economics are different (more chambers, longer test times per unit) and the addressable market is multi-billion incremental on top of the core ATE TAM. Teradyne explicitly anchored its $6B revenue target model around the “wafer-to-AI-datacenter” framing. Advantest is positioning through M&A in system-level and silicon validation.

• Silicon photonics and co-packaged optics (CPO). As Nvidia, Broadcom, and the hyperscaler ASIC supply chain move to optical interconnects between accelerators and switches, every photonic die requires both electrical and optical test — a category that didn’t materially exist three years ago. Advantest received its first high-volume ATE order for silicon photonics in CY25. Teradyne launched the Photon100 platform specifically for this.

• Probe cards, contactors, and test interface boards. Teradyne owns ~10% of Technoprobe. Cohu and FormFactor sit alongside as picks-and-shovels exposure.

• AI-driven adaptive test. Uses ML to reduce test time on known-good silicon. Counterintuitively margin-positive for the duopoly because it sells as software-attached value on the same hardware base.

Outlook and risks

Capacity is the binding constraint, not demand. Advantest is ramping to 10,000 SoC test systems per year — up from a 7,500 target — to keep pace with a known product pipeline: Blackwell Ultra (B300) and AMD MI400 in 2026, custom AI silicon from Broadcom, Marvell, and Alchip stacking through OSAT lines, and HBM4 (16-Hi, 2,048-bit) qualifying for production. Each generation lengthens the test program. Teradyne is signaling 55–60% first-half revenue linearity — the order book is already loaded.

The market backdrop is just as durable. Advantest forecasts the SoC tester market at $8.7–9.5B in CY26 versus $6.9B in CY25, and the merchant AI ASIC supply chain (Broadcom-Google, Marvell-Meta/Microsoft, AWS Trainium) keeps multiplying tester demand because each new custom silicon program qualifies a fresh test program. The four adjacent product surfaces above — SLT, photonics, probe cards, adaptive test — extend the runway by another half-decade beyond the core ATE box.

Risks live on the product roadmap, not the financial model. In-sourcing by Nvidia or TSMC has been explicitly rejected on engineering ROI grounds. Chinese indigenous ATE remains generations behind in leading-edge SoC. Customer concentration in the AI OSAT supply chain — most acute for Advantest — is the largest single tail risk; a hyperscaler capex digestion cycle is the scenario worth tracking into 2027.

Net: the cleanest non-Nvidia exposure to AI accelerator unit volume — a duopoly with order-of-magnitude growth in test surface area, three multi-billion adjacent product surfaces, and the engineering switching costs to defend share through the next decade..

The thing that collapsed is called the KV cache. Understand how it works, and the next five years of AI economics, hardware roadmaps, and infrastructure margins start to make perfect sense.

Two cost lines

Running an AI model (inference) comes down to two cost lines:

• The Model Weights (Fixed Cost): These are the model’s “brains,” paid for during the massive training runs and amortized across every request the model ever serves. With enough traffic, weights are essentially free.

• The KV Cache (Variable Cost): This is the model’s “short-term memory.” It is paid for in High Bandwidth Memory (HBM)—the ultra-fast, ultra-expensive RAM that sits right next to the GPU. This cost is incurred per user, per word, on every single request. It does not amortize. It scales linearly.

This second line dictates the unit economics of AI. It is also the line frontier labs talk about least, because it destroys their published “price-per-token” the moment a user’s prompt gets too long.

What the cache actually is

Transformers generate text one word (token) at a time. To write the next word, the AI has to look back at every previous word in the conversation. Re-reading the entire conversation from scratch every single step would be incredibly slow, so the model saves its notes in a “Key-Value” (KV) cache in the GPU’s memory.

This cache grows with every single word you type or the AI generates. For a large model like Llama 3 70B, a 128,000-token conversation eats up ~40 GB of memory per user. Multiply that by concurrent users hitting the same server, and you hit a memory wall long before you run out of raw computing power.

The bottleneck is bandwidth, not compute

People who grew up in the software era expect processor speed (compute) to be the bottleneck. It isn’t.

Modern GPUs like the NVIDIA H100/B200 are incredibly fast calculators, but their memory bandwidth can’t keep up. The math processors spend most of their time sitting idle, waiting for the short-term memory (the KV cache) to be fetched and delivered.

The math is brutal. When generating a response for a long document, the GPU is essentially reduced to a giant memory streamer with a math coprocessor stapled on. The speed limit is entirely dictated by how fast data can move through the HBM. This is why every 2026 accelerator roadmap is aggressively focused on memory bandwidth, not raw processing speed.

How Long Context Broke the Pricing Model

AI companies publish a single price per million tokens. That price is modeled against short interactions (4K to 8K tokens) where the KV cache is small enough to be a rounding error. Run that same model at 128K tokens, and the cost structure is completely unrecognizable.

Walk through the numbers: An H100 GPU holds the model’s weights and uses whatever memory is left over for the users’ KV cache. At an 8K context, a server might support ~190 concurrent users. At a 128K context, it supports one or two. It’s the same GPU, running the same model, charging the same nominal price—but the actual cost to the provider just skyrocketed.

This is why providers quietly changed their pricing in 2025. “Cached” tokens—where short-term memory is reused across similar requests—now cost a fraction of fresh tokens. The discount looks generous to developers, but it’s actually a confession from the AI labs about their underlying cost curve.

Agents will make this worse. Autonomous AI agents don’t have clean, reusable memory. They constantly replay long, mutating histories as they use tools and browse the web. Anyone modeling enterprise AI margins based on simple chatbots is in for a shock when traffic shifts to agents.

Architecture is outrunning hardware

Look at where the recent gains came from.