Bottom line

1. GPU debt went from ~14% private credit in 2023 to A3 / SOFR+225 in 2026. More than $20B of GPU-backed facilities have been announced.

2. The market learned that the real collateral is contracted cash flow. CoreWeave can borrow at +225 against a hyperscaler and +450–550 against weaker customers, using broadly the same hardware.

3. The system looks diversified but may not be. Customers prepay, lenders finance the middle, and Nvidia and Google increasingly insure the tail. Yet all of them ultimately depend on continued AI demand.

Note: Special thanks to Brexton Pham (Co-head of Compute at Cantor) for key insights described below.

The founding loan

On August 3, 2023, CoreWeave closed a $2.3 billion delayed-draw term loan co-led by Magnetar, an Evanston fund with two decades of structured-credit expertise, and Blackstone Tactical Opportunities, with Coatue, DigitalBridge, and BlackRock, PIMCO, and Carlyle funds in the syndicate. Back-of-envelope, at the reported ~70¢ advance rate, the collateral pool was $3B+ of H100 servers, on the order of 70–100k GPUs.

Hardware-secured lending wasn’t new, strictly speaking. Upper90 had been writing equipment-backed facilities for Crusoe since 2019, and the crypto cycle produced plenty of smaller loans against mining hardware. What was new in August 2023 was the scale and the structure. Nobody had borrowed billions against accelerators, and nobody had paired the chips with assigned customer contracts inside one bankruptcy-remote SPV. That pairing is the invention, and every GPU deal since is a descendant of it.

The borrower hardly looked like institutional-grade credit. Five years earlier CoreWeave was Atlantic Crypto, an Ethereum miner run by three former commodities traders. Magnetar was in first, and early: $50M of convertible notes in November 2021, two years before large credit funds touched the space. Blackstone arrived with the debt, co-leading once there were contracts to lend against. By mid-2023 Nvidia was on the cap table, Microsoft was becoming the dominant customer, and CoreWeave faced a capex problem equity could no longer solve. A single H100 data-center build could cost more than the company had raised in its lifetime. (The conviction paid historically well. That $50M note converted at the IPO, and by late 2025 Magnetar’s stake was worth roughly $12.5B, one of the defining credit-to-equity calls of the AI cycle.)

Priced at a reported 11–14% all-in (SOFR was ~5.3%, so call it 600–870bps of spread) for senior secured paper, the deal looks expensive in hindsight and wasn’t. Magnetar and Blackstone were underwriting a structure with no template, so they wrote one. The SPV design, the contract assignment, the depreciation-paced amortization: every deal since traces to choices made in this document. What made it writable at all was take-or-pay contracted cash flow from an investment-grade counterparty, sized to fully amortize the loan before the chips age out. The H100’s spot scarcity, trading above list with long waiting lists, was the recovery story, not the credit. That design choice, repay before you ever have to sell a used GPU, is the founding insight of the whole asset class, and it has never changed.

The repricing

CoreWeave inadvertendly created a controlled experiment in GPU credit. Hold the borrower and hardware roughly constant; vary the customer and the contract. The spread moves by hundreds of basis points. Across the eight facilities you can watch the collateral concept itself migrate: from the GPU, to the contract, to the counterparty.

Each facility resolved a specific question. DDTL 2.0 in May 2024, $7.5B with the same leads plus Carlyle, CDPQ, and Eldridge, answered whether the structure could scale. The October 2024 bank revolver (JPM, Goldman, Morgan Stanley) answered whether bank credit committees would touch the collateral. The May 2025 unsecured notes at 9.25% ran the control: unsecured CoreWeave priced wider than GPU-secured CoreWeave, the market’s way of saying the SPV structure itself carries several hundred basis points of value.

DDTL 4.0 in March 2026 is the one to study. $8.5B at SOFR+225 (5.9% fixed tranche), rated A3/A(low), structured by MUFG and Morgan Stanley, anchored by Blackstone Credit & Insurance. That anchor puts Blackstone at every stage of the curve. Tactical Opportunities underwrote the trade when it was unproven in 2023, stayed lead as it scaled to the $7.5B DDTL 2.0, among the largest private credit financings ever completed, and the firm’s credit and insurance arm anchored the investment-grade version three years later. Few firms get to create an asset class, scale it, and then become its senior buyer, and it took a platform of Blackstone’s breadth to do all three from one shop. Two things changed under the hood. The collateral narrowed to one SPV and its associated hyperscaler contract, so Moody’s was effectively rating the counterparty, not the chips. And the lenders themselves moved the borrowing base away from hardware value and toward contract execution.

Two facilities later the curve had a full term structure: DDTL 5.0 (two non-IG customers) at SOFR+450, Ba2/BB+; DDTL 5.5 (three-year contracts, re-lease optionality) at SOFR+550. Same borrower, same silicon, 325bps between tiers. The market is overwhelmingly pricing offtaker credit and contract tenor, not GPU residual value.

The curve is already being replicated across borrowers. CoreWeave’s A3 paper at +225 pays roughly double what generic single-A corporate credit has traded this cycle, a premium for structure novelty and renewal risk, and the reason insurance accounts showed up. In July, Nebius raised $775M of secured debt at SOFR+250 from a ten-bank MUFG-led group on its first secured deal. In August, Lambda closed a $926M term loan B at SOFR+300, rated Baa2, fully amortizing to 2030. The operative phrase in Lambda’s own announcement is “an investment-grade offtaker.” Three borrowers, one rate card. The compression is worth real money too: on DDTL 4.0’s $8.5B, the difference between 2023’s ~870bps and today’s 225 is roughly $550M a year of interest expense.

The rate card has a floor, though, and it’s worth being concrete about who lives under it. Price a modest cluster: 1,024 GPUs at $3.25/GPU-hour on a four-year committed contract is about $117M of total contract value. At the 25–30% downpayments unrated borrowers face, that means roughly $30M of cash out the door before the first token is served. The compressed spreads above go to whoever brings an investment-grade contract; a small neocloud without one still sees quotes reported in the SOFR+800–900 range. A startup that can fill nodes but can’t sign a hyperscaler pays roughly what CoreWeave paid before the asset class existed.

FIG. 2 — Spread over SOFR by facility. Bubble area is facility size; dashed rings are spreads estimated from reported all-in cost.

The template travels

The CoreWeave template got copied, then it mutated. Four variants, each changing something specific in the financial architecture.

Direct replication proved the template travels across borrowers: Lambda’s $500M Macquarie facility (April 2024), Crusoe’s ~$425M via Upper90, Fluidstack’s Macquarie facilities of up to $10B in capacity, Nebius against its Microsoft offtake. Macquarie franchised the model to the second tier while Blackstone and Magnetar stayed concentrated in the leader.

The asset-owner SPV separates the GPU-owning vehicle from the compute customer entirely. xAI’s Colossus 2 financing has a standalone SPV raising ~$20B to buy the GPUs and lease them to xAI, with Nvidia putting up to $2B of equity into the vehicle buying its own product. The compute never touches xAI’s balance sheet. It’s aircraft leasing with a five-year asset instead of a twenty-five-year one, minus the half-century of residual-value data that makes aircraft leasing work.

Powered-land financing brings in the Bitcoin miners, who contribute the scarce inputs of power and grid interconnects rather than GPUs. TeraWulf signed 10-year hosting agreements with Fluidstack worth $3.7B, with Google backstopping $1.8B of the lease obligations for warrants on ~8% of the company; Cipher ran the same trade a month later ($3B, $1.4B backstop, ~5.4% stake). Cantor Fitzgerald’s lane is here rather than in the term-loan syndicates: as the conversion trade’s house bank, it runs the collateralized-lending playbook it built for Bitcoin one asset class over.

Vendor credit support may be the most consequential mutation. Nvidia guaranteed $6.3B of CoreWeave’s unsold capacity through 2032, then in August signed MOUs with Apollo, BlackRock, Blackstone, Brookfield, Goldman, and KKR for financing platforms targeting $500B, keeping an option to backstop $125B of it. Add Google’s miner backstops and the pattern is clear: the companies with the best information about GPU demand have started selling credit support instead of deploying capital, supporting demand without funding the capex themselves. A backstop costs nothing today, sits off balance sheet, earns warrants or ecosystem lock-in, and becomes real only in the states of the world where the guarantor’s own product has failed. It is brilliant right up until it is correlated.

FIG. 3 — Announced GPU-backed facilities by borrower (capacity, not drawn balances).

The risk stack

Put the pieces together and the risk sits in four layers.

FIG. 4 — The risk stack and the loop underneath it. Distributed across five balance sheets; correlated to one variable.

Customers take the first loss without calling it that: per the S-1, CoreWeave offtakers prepay 15–25% of contract value at signing on take-or-pay commitments covering 96% of revenue, functionally the equity tranche of their supplier’s capex. Private credit and banks hold the senior middle, protected by contract cash flow and amortization. Insurance holds the duration: the A3 rating exists because general accounts (Apollo’s Athene, KKR’s Global Atlantic, Brookfield’s Wealth Solutions, Blackstone Credit & Insurance anchoring DDTL 4.0) mostly can’t buy sub-IG paper, and Meta’s Hyperion SPV ($27B of A+ debt, PIMCO holding ~$18B) proved the trade at scale. And the vendors sell protection on the tail.

The architecture distributes risk beautifully across legal entities. It does not necessarily diversify the economic driver. Follow the money in a circle: an AI customer raises capital, and prepays for compute. The neocloud borrows against that customer’s contract. The lender funds the GPU purchase. Nvidia books the revenue. Nvidia then backstops future capacity, because it wants the next round of purchases funded. Five balance sheets, four layers of protection, one loop. Every layer is underwriting the same assumption: that AI demand keeps compounding.

The diversification is nominal. The correlation is one.

What a used GPU is actually worth

The entire structure is built to avoid answering this question, and you can prove it from the documents. I ran a term search of CoreWeave’s DDTL 5.5 credit agreement, the newest one on file. “GPU” appears dozens of times, in defined terms like “GPU Clusters” and “GPU Depreciated Amount.” The words a lender would use to value hardware against an outside reference appear zero times: no appraisal, no residual value, no orderly liquidation, no advance rate, no loan-to-value, no borrowing base, no remarketing. An $8.5B-scale lending program, secured by chips, with no mechanism anywhere for asking what the chips would fetch.

The sharpest tell comes from Nvidia itself. In August it filed a Residual Value Guaranty covering up to $105B on the ~4.25GW of data centers SB Energy is building for OpenAI, which sounds like the vendor finally putting a floor under its product. Read the exhibit. The guaranteed minimum value is defined as data-center, power, and transmission-related costs, and Nvidia’s own equipment is carved out as “Guarantor Property” under a separate repossession clause. The company with the most information about GPU value on earth will guarantee the building and the substation. It will not put a number on the chips.

Meanwhile, what we actually know is thin. Rental rates round-tripped violently, from ~$8/hour in early 2024 to $1.70 by October 2025 and back to ~$2.35 by March 2026. Depreciation schedules disagree by 50% on identical hardware (CoreWeave six years, Nebius four). The broadest public record of executed used-GPU sales, CCIR’s three-year dataset covering the datacenter models it tracks, comes to 3,009 listings, 10,911 units, $26.3M; bilateral trades outside that universe go unrecorded, which is itself part of the problem. Manheim, the used-car benchmark lenders actually underwrite against, processes over 5 million transactions a year. Three years of the whole used-GPU market is about 0.3% of one CoreWeave facility.

And “the GPU” undersells what is actually on the balance sheet. What lenders finance is machines: eight-GPU servers with CPUs, memory, NICs, and switching wrapped around the accelerators, deployed in a specific network topology in a specific building. A machine carries at least three different residual values. There is the face value its depreciation schedule implies; the going-concern value if a buyer takes over the cluster running, which secondary-market estimates put around 50–70% of new for 2–3-year-old hardware; and the orderly-liquidation value if it has to be de-installed, re-certified, and parted out, plausibly 30–50% in a distressed sale, before the cost of redeploying racks engineered for someone else’s data center. Hardware wear is real at fleet scale (Meta logged 419 unplanned disruptions across 16,384 H100s in 54 days of Llama 3 training), so condition varies machine by machine. Today’s structures mostly don’t distinguish among the three values. The one that matters in a default is the last, and it is the one with the least data behind it.

A market is now being built, but ends-first. Silicon Data publishes a daily H100 rental index with a 36-month forward curve, and in August launched a GPU residual value product, a DCF off its own rental curve. That is a value, not a price: it tells you what a GPU earns for whoever keeps running it, not what a lender recovers when they have to sell one. Compute Exchange opened a used-GPU marketplace that publishes reference quotes but takes no custody and clears no trades. CME lists cash-settled H100 and B200 rental futures on October 5, pending review, while the CFTC’s open request for comment questions whether compute yet has the fungibility, standardization, and liquidity to underlie derivatives at all. Even once the futures trade, they hedge the hour, not the box: generation, condition, location, and redeployment cost all sit in the basis between a rental rate and a rack, and the rack is what the lender owns after a default.

The bears (Chanos, Burry) argue two to three years of true economic life. A collateral class with indexes but no prints can’t prove them wrong.

When the contracts expire

The irony is that GPUs became credible collateral only after lenders learned how not to rely on their collateral value. Customer prepays provide a first-loss cushion. Take-or-pay contracts turn uncertain utilization into scheduled cash flow. Loans amortize before the hardware is expected to become obsolete. Investment-grade offtakers collapse spreads. Nvidia and Google increasingly insure the tail. Each innovation pushes the moment when someone has to answer the uncomfortable question of what a three-year-old GPU is actually worth further into the future.

That is why the 2026–27 renewal cycle matters more than any depreciation schedule. The 2023–24 contract vintage starts rolling off next year. If the first generation of contracts renews, lenders may never need a robust answer, and GPU finance will look less like equipment lending and more like project finance wrapped around compute contracts. If they do not renew, the entire structure falls back onto a secondary market where the publicly documented transactions total just $26.3M.

And that exposes the larger contradiction. The risk has been distributed across customers, neoclouds, banks, insurers, and vendors, but every layer ultimately rests on the same underlying assumption: AI demand remains strong enough to keep the machines earning.

The GPU became collateral because finance learned how to avoid testing the collateral. The first real test begins when the contracts expire.

Sources & notesCompany releases and SEC filings via CoreWeave IR (including the DDTL 5.5 credit agreement), Nvidia’s Residual Value Guaranty exhibit, Lambda, Nebius, and TeraWulf; CCIR on used-GPU transactions; CME Group / Silicon Data; Data Center Dynamics; PR Newswire; Bloomberg; Nvidia Newsroom; Fortune; Friedman (2026); Steffen (2026); CipherTalk on residual values. DDTL 1.0/2.0 spreads are estimates from reported all-in cost versus prevailing SOFR. Sector total reflects announced facility capacity, not drawn balances. · datagravity.dev

The NVL72 has become one of the basic units of account for Blackwell-scale AI infrastructure — capacity deals, neocloud debt, and NVIDIA revenue models are increasingly denominated in racks. Yet the most-quoted number, “a $3 million rack,” described GB200 and answers the wrong question. The anchor here is a set of recent purchase orders for GB300 NVL72 racks reviewed by Data Gravity in August 2026 (figures adjusted modestly to preserve the buyer’s anonymity), cross-checked against analyst teardowns and public specs.

What you’re buying

A supercomputer sold as one line item: 72 Blackwell Ultra GPUs and 36 Grace CPUs in 18 liquid-cooled compute trays, fused into a single coherent domain by 9 NVLink switch trays — 130 TB/s of aggregate bandwidth. 288 GB of HBM3e per GPU, ~21 TB per rack. It draws 132–140 kW, ten times a conventional enterprise rack, is 100% liquid-cooled, and ships crated at nearly 4,000 lbs.

The rack: about $5 million

Media estimates put GB300 NVL72 at $6–6.5M reported. The best evidence sits lower: recent purchase orders put it at just under $5.0M without an in-rack CDU, and ~$32K more with a 250 kW CDU PO — real transacted numbers for small orders, not list prices. Hyperscale volume buyers will do better. That’s ~$69K per GPU slot, versus ~$40–50K reported for the GPU module alone; the premium is the rest of the machine.

Compute trays are ~87% of the value; split them open and the 72 GPU modules alone are ~$3.4M, with the Grace complex, NICs, and tray hardware making up the rest. With the NVLink fabric included, NVIDIA content exceeds 92%. Everything an infrastructure investor pictures as “the rack” — power shelves, busbar, cold plates, steel, cabling, integration — is under 8%. But it’s the fastest-growing 8%: Morgan Stanley pegs in-rack cooling content up ~20% per generation ($42K on GB200 → $50K on GB300 → ~$56K on Vera Rubin), and rack power keeps climbing toward Rubin-class ~250 kW — a structural tailwind for liquid-cooling, power-electronics, and HBM suppliers even as NVIDIA takes the headline dollars.

Deployed: closer to $5.7 million

A purchased rack is EXW — your problem from the loading dock. Making it produce tokens in an existing AI-capable facility adds five line items (deliberately excluded: datacenter construction, cluster-level core switching, storage — campus decisions, not per-rack ones):

Operating: $240–410K a year

The three price points are anchored, not arbitrary: $0.05/kWh approximates a strong hyperscale PPA in cheap-power markets, $0.10 sits just above the U.S. industrial average of 9.2¢ (EIA, June 2026, up 3% year over year), and $0.15 covers constrained or commercial-rate markets. Power is real money but not the story: even at $0.15/kWh, five years of electricity (~$1.1M) is under a fifth of deployed capex; at colo rates, the five-year facility bill is 18–26% of total cost. The NVL72 is capex-dominated — utilization and depreciation assumptions, far more than power prices, decide whether AI compute is profitable. Facility cost is meaningful but depreciation dominates: roughly $12–34K a month for the facility versus $95–158K a month of straight-line hardware depreciation.

What a GPU-hour needs to cost

One rack holds 72 × 24 × 365 = 630,720 GPU-hours a year; at 60/80/90% utilization you monetize 378K / 505K / 568K of them. We amortize the $5.68M straight-line over 5 years — the center of hyperscaler practice (Microsoft, Google, and Oracle depreciate AI servers over 4–6 years; Amazon runs closer to 4–5; CoreWeave uses 6) and consistent with the rack’s 3-year warranty plus extensions. Add ~$318K/yr of colo:

At 80–90% utilization the rack produces compute at $2.56–2.88 per utilized GPU-hour; let utilization slip to 60% and the same rack costs $3.84. Utilization, not the amortization debate, is the bigger lever — and both matter less than the rental price the market will bear (GB200-class capacity has listed around $10.50/GPU-hr, with committed deals well below). One honest caveat on the 5-year life: hardware runs in year 5, but rental rates decay with each GPU generation, so revenue-weighted life is shorter than accounting life.

Why the headline price misleads

A $5 million rack is not a $5 million decision. It’s a multi-year commitment to ~140 kW of liquid-cooled power, several hundred thousand dollars of network attach, and — above all — enough demand to keep 72 GPUs busy before the next generation resets the economics. The purchase price of an NVL72 tells you surprisingly little about the cost of the compute it produces. NVIDIA’s ~$5M of hardware sets the level of those economics; your utilization and useful-life assumptions decide whether they work.

Sources & method

1. Recent purchase orders for GB300 NVL72 racks (two configurations, with/without in-rack CDU) and accompanying rack BOMs, reviewed by Data Gravity, August 2026. Figures adjusted modestly to preserve the buyer’s anonymity.

2. Tom’s Hardware — Vera Rubin NVL72 pricing; GB200 $2.8–3.4M and GB300 $6–6.5M reported; compute trays ~90% of system cost

3. Tom’s Hardware / Morgan Stanley — GB300 NVL72 liquid-cooling content ≈$49,860/rack, +20% vs GB200

4. Supermicro GB300 NVL72 datasheet — 21 TB HBM3e, 132–140 kW, CDU options

5. NVIDIA GB200 NVL72 specifications · Lenovo GB300 NVL72 product guide

6. Tae Kim (Barron’s) — GB200 NVL72 ≈ $3M per rack

7. Encor Advisors — North American wholesale colocation ≈$196/kW-month, H2 2025

8. EIA Electricity Monthly Update — U.S. industrial average 9.17¢/kWh, commercial 14.19¢/kWh (June 2026)

9. Spheron — GB200 NVL72 rental from ~$10.50/GPU-hr

10. Yardeni Research — AI server depreciation schedules across hyperscalers (4–6 years; CoreWeave 6)

No architecture is cheapest everywhere: cost per million tokens against tokens per second per user, with measured crossover points between architectures

In February, AMD’s MI355X served Qwen3.5 at 192 tokens per second per GPU. In May, the same GPU served the same model at 3,660. Nothing shipped from a fab in between; three pull requests landed in an open-source inference engine, and the chip got nineteen times faster.

Around the same time, SemiAnalysis measured a B200 serving DeepSeek R1 at 941 tokens per second per GPU in a standard 8-GPU server, and 4,130 when the identical die sat in a 72-GPU NVLink rack. Same silicon, same model, 4.4x apart.

What kind of product behaves like that? Not a chip. A chip’s performance does not quadruple with its packaging or multiply nineteen-fold with a software release. A system’s does. That is the argument of this piece: the accelerator cannot be evaluated apart from the system around it, because the system, not the die, sets cost per token. Most accelerator pitches I see still lead with a FLOPS bar chart. The measurements above are what that chart cannot see, and the end of this piece is a picture of the market they are quietly building.

So here is the claim in its strongest form. Training was one optimization problem, homogeneous enough that the fastest chip in the tightest cluster won nearly every time, and it built homogeneous fleets. Inference is many problems wearing one name. Models vary a hundredfold in size, contexts vary a thousandfold in length, some buyers need a token every seven milliseconds and some need a million tokens for a nickel, and prefill and decode stress opposite parts of the machine inside a single request.

Against that variety, each accelerator traces out a performance surface over the space of workloads, and the surfaces cross. There is no fastest chip, only chips that are fastest somewhere. Heterogeneous workloads make heterogeneous fleets, which is why Anthropic already serves Claude on TPUs, Trainium, and NVIDIA GPUs at once, and why the interesting question is no longer whether inference goes multi-silicon but who besides the hyperscalers gets to benefit.

Same silicon, different system: measured throughput multiples on identical or equal-FLOPS silicon when one part of the system changes, from 1.9x for bandwidth to 19x for software

By the numbers: the ridge point band, the H100 to H200 gain with zero extra FLOPS, same-chip fabric gains, software speedups on fixed silicon, the memory wall ratio, and the ASIC share of 2026 AI servers

How a Token Gets Made

To see why the FLOPS model fails, follow one request through the machine. It has two phases. First the model reads your whole prompt in a single parallel pass, called prefill. This is the compute-heavy part, and it is the phase GPU marketing implicitly describes. As it reads, the model writes itself notes about every token, called the KV cache, so it never has to reread the raw text.

Then comes decode, the phase that produces every visible token, one at a time. To emit each one, the GPU must stream every active model weight and the session’s entire cache out of memory, run a small amount of arithmetic on the way through, and repeat. At small batch sizes, decode performs about one or two floating-point operations per byte it loads, against the two to three hundred per byte the chip was built for. Until batching fills them, the math units spend most of the token-generating phase waiting on bytes.

That imbalance is not an accident of one design; it is decades of hardware history. Peak compute grew roughly 60,000x over two decades while memory bandwidth grew 100x, and the gap widens every generation: from A100 to Rubin, low-precision compute rose at least 56x against 11x for bandwidth. Training escapes the imbalance by batching enormously. Inference is where it gets billed.

The memory wall: compute throughput has outrun memory bandwidth every generation from A100 to Rubin, while the ridge point that decides whether decode is compute- or bandwidth-bound has barely moved

Hold that picture, a token is a memory read with some math attached, and the rest of the piece is a tour of the places bytes get stuck: the accelerator’s own memory, its bandwidth, the storage behind it, the wires between chips, and the software that orchestrates all of it. At every stop the ranking of chips can flip, and that is not a figure of speech. The reversals below are all measured.

Fit Beats FLOPS

The first constraint is the dumbest: does the model fit? Llama 405B at full precision needs about 1,053GB with working overhead. A node of eight H100s holds 640GB, so the model must be quantized or split across two servers, with every decode step paying the network toll between them. Eight MI300Xs hold 1,536GB. The model simply fits, with 480GB to spare.

The spare room matters more than it looks, because leftover memory holds the KV caches of concurrent users, and concurrency is what fills idle math units. A chip that fits more sessions runs bigger batches, and bigger batches convert wasted FLOPS into billable ones. When dstack benchmarked 405B in late 2024, with AMD’s software still visibly immature, the MI300X node produced roughly 1,970 tokens per second against 643 for the H100 node, which had no memory left to batch with. Three to one, for the chip with the weaker software and the lower street price. Capacity is compute you get to use.

Memory budget: Llama 3.1 405B at FP16 needs about 1,053GB with overhead; a node of 8 H100s holds 640GB and must quantize or split, while 8 MI300Xs or 8 H200s fit it whole

The H200 Experiment

If decode really is a memory read, a clean prediction follows: add bandwidth to a chip without adding compute, and tokens per second should rise anyway. NVIDIA ran that experiment and sold it as a product. The H200 is the H100’s exact die with more and faster memory attached, not one additional FLOP, and it delivers up to 1.9x the throughput on Llama 70B. Time to first token barely moved, because prefill is the compute-bound phase and prefill got nothing.

Rank today’s accelerators by measured decode ceiling and the order follows the bandwidth column of the spec sheet, not the FLOPS column. Push the logic to its limit and you get the strange machines at the market’s edge: Cerebras serves Llama 405B at 969 tokens per second per user from a wafer holding just 44GB, because that 44GB sits behind 21PB/s. And you get the strangest data point of all: in December 2025, NVIDIA licensed Groq’s SRAM-based architecture in a deal reported near $20B, and it now markets the result, the Groq 3 LPX rack, as a decode companion to Vera Rubin. The company with the most FLOPS in the world buying a bandwidth machine tells you which resource decode actually consumes.

Tokens per second follows bandwidth, not FLOPS: the H200 has 1.0x the H100’s compute and 1.4x its bandwidth and delivers up to 1.9x the throughput; across five accelerators the decode ceiling tracks bandwidth

Agents Changed the Bill

The number I keep coming back to is 126,000. That is the median prefix, in tokens, that a real coding agent reads per LLM call, measured across 4,265 sessions in TraceLab’s study, to produce about 252 tokens of output. The 90th percentile prefix is 467,000 tokens, roughly 149GB of KV cache on a Llama-class model. More than any accelerator sold, for one user’s session, and every constraint so far, fit and bandwidth alike, scales with exactly this variable.

That shape is money. At a typical $3-per-million input rate, rereading a 126K prefix costs about 38 cents per call, hundreds of times per session; served from cache at the standard 90% discount, about 4 cents. So the ranking of chips became a function of context: at 4K tokens the contest is bandwidth per dollar, at 128K it is who fits more users in memory, and at agentic scale the cache does not fit in memory at all. At that point the accelerator’s most important competitor is not another accelerator.

KV cache versus context length, log scale: five current model architectures diverge nearly 10x in cache per token, and by one million tokens even the most efficient exceeds a B200’s capacity

Storage Becomes Compute

It is the storage tier. Once inference reuses state, storage and memory become substitutes for compute: a prefix the system has seen before can be recomputed on a $30,000 GPU or read back from flash, and reading wins by an order of magnitude. The 90% cached-token discounts at OpenAI, Anthropic, and Google are that substitution, printed on a price sheet. DeepSeek, which has kept its cache on SSDs since 2024, discounts 97%.

The engineering results say the discounts are economics, not marketing. Lablup and VAST cut warm-turn time to first token on 140K-token agent contexts from 22.0 seconds to 6.6 by serving the cache from external flash; LMCache took Qwen3-235B from 3.98 seconds to 0.29 with a host-DRAM tier. Identical GPUs before and after. TrendForce attributes the record $18.5B enterprise-SSD quarter in early 2026 directly to AI agents, and NVIDIA now publishes a four-level hierarchy telling operators where to park state, from HBM down to networked flash. When the GPU vendor writes a storage architecture, storage has joined the inference system.

The Fabric Tax

Frontier models stopped fitting in one server, so every decode step now includes a conversation between chips, and the price of that conversation depends on which side of a cliff you stand. Inside a rack-scale domain, a GB200 GPU reaches any of 71 peers at 900GB/s. The moment traffic leaves an 8-GPU box for ordinary 400G networking, the figure is about 50GB/s. Eighteen times apart, and invisible on any spec sheet, because it is a property of the wiring, not the chip.

The 4.4x from the opening is this cliff, measured. Mixture-of-experts models like DeepSeek R1 scatter hundreds of experts across GPUs and shuttle every token among them mid-inference; the rack’s fabric makes that cheap, the box’s fabric makes it ruinous, and in SemiAnalysis’s like-for-like disaggregated comparison, built to isolate exactly this variable, the same B200 die lands at 4,130 versus 941 tokens per second per GPU. The corollary explains NVIDIA’s own behavior better than any keynote: a company that once sold cards now sells 72-GPU domains, and its networking business booked $14.8B last quarter, up 199%. The wires became part of the accelerator.

Same chip, different fabric: B200 in an 8-GPU HGX node versus GB200 NVL72 on DeepSeek R1 FP4; the tokens-per-GPU gap widens from 1.7x to 4.4x as interactivity rises

The Software Month

Which leaves the 19x, and its resolution is the most unsettling of all, because the component that moved was invisible. The three SGLang pull requests behind the MI355X’s jump were a fused expert kernel, a rewritten normalization path, and a fused top-k. That is the entire changelog between a chip that looked uncompetitive in February and one that looked excellent in May.

This is not an AMD story. The B200 got five times faster on gpt-oss-120b in two months as new quantization and speculative decoding landed, taking NVIDIA’s own cost per million tokens from $0.11 to $0.02. Meanwhile the H200 running TensorRT-LLM did not move for four months; that pairing had already reached its ceiling. So software gains are enormous, uneven across vendors, and unpredictable in timing, and two conclusions follow. Software maturity deserves a column on the spec sheet, where it is the most defensible thing CUDA has. And every benchmark is a snapshot of hardware plus that month’s software. A ranking from October is not a ranking in May.

Same silicon, different month: measured tokens per second per GPU on fixed hardware before and after software releases, log scale

There Is No Fastest Chip

Assemble the whole machine. Fit, bandwidth, state, storage, fabric, and software each move throughput by integer multiples, each binds a different workload, and none of them appears in a FLOPS comparison. FLOPS still matter; they are one constraint among six. The precise claim is that peak FLOPS lose their explanatory power the moment any other constraint binds, and in inference another constraint almost always binds. A workload is a point in a space; each accelerator is a surface over that space; the surfaces cross. In the current benchmark record alone: MI325X undercuts H200 on Llama 70B cost at every interactivity level, MI355X undercuts B200 on gpt-oss below 225 tokens per second per user and loses above it, and the rack-versus-box crossover moved between rounds because the software on each side changed at different speeds.

The accelerator map: current inference silicon compared on dense FP8 compute, HBM capacity, bandwidth, scale-up domain, software maturity and best-fit workload region

Two facts, then, and everything from here follows from them. Performance is a property of the system, so no single architecture should be expected to dominate every workload. And inference is becoming stateful, so the workloads themselves are diverging: chat, agents, batch, and latency traffic now stress the machine in different places. Put the two together and the familiar question dissolves. Asking which inference chip is best is like asking which aircraft is best. A 787 and an A321 are not competitors; they are answers to different routes, and no serious airline runs one airframe. Carriers match aircraft to routes, obsess over load factor and fuel, and treat the fleet mix itself as the strategy. Load factor even has an exact analog here: batch utilization, the share of a chip’s paid-for compute that passengers actually fill. Inference compute is becoming an airline business, and almost everyone is still shopping for the one best plane.What the Fleet Looks Like in 2029

Walk into a large inference facility in 2029, as I expect it to be built, and the GPU warehouse is gone. One hall runs rack-scale domains, NVLink or its Ethernet rivals, serving the frontier mixture-of-experts models that need seventy chips talking at 900GB/s. The next hall is high-memory nodes packed with long-context agent sessions, chosen because fit beats FLOPS for that traffic. A third belongs to co-designed ASICs earning their keep on the narrow, stable set of workloads they were optimized around. Near the front, where the latency-critical traffic lands, a few aisles of SRAM machines produce tokens faster than a person can read them, the corner Cerebras’s 969 tokens per second per user already stakes out. And underneath every hall, the quietest and maybe most valuable layer in the building: a flash estate holding the accumulated context of millions of agents, exabytes of KV cache that get read back at 90 to 97% below the price of recomputing them.

Every element of that building is shipping or announced today. The mixed halls are Anthropic’s fleet: up to a million TPUs, over a million Trainium chips, NVIDIA capacity through the clouds, one model family across all of it. The commitments are OpenAI’s: ten announced gigawatts with NVIDIA, six with AMD, ten of Broadcom custom silicon, roughly two of Trainium, multi-year announced figures rather than deployed capacity, but a fleet plan by any name. The ASIC hall is Microsoft’s Maia 200 serving GPT-5.2 and Copilot. The flash estate announced itself in TrendForce’s numbers as a record $18.5B enterprise-SSD quarter attributed to AI agents. Even NVIDIA is building the mixed hall internally: by its own product materials, the Vera Rubin platform spans Rubin GPUs, the Groq-derived LPX decode rack, BlueField in front of storage, a Vera CPU underneath. ASIC-based systems are already 27.8% of 2026 AI server shipments and growing faster than merchant GPUs. The 2029 building is just 2026 with the ratios moved.

Now price the building. The airframers each keep the routes they are shaped for, and NVIDIA’s route is the biggest: the broad, changing middle of the workload space, where a buyer cannot predict its own mix and CUDA’s maturity is worth a real premium. Data center revenue of $75.2B last quarter, up 92%, is not a franchise in decline. But the margin structure of the building favors a different class of company, the ones every hall depends on no matter whose logo is on the racks. The serving engines that let one model run across five silicon families. The KV-cache storage tier that turns recompute into a read. The fabrics, NVLink on one side and the UALink and Ethernet coalition on the other, that set how large a coherent domain can get. Airlines fought each other to thin margins for a century while the airports, the leasing companies, and the reservation systems compounded. The picks and shovels of a fragmenting market are the layers that make the fragments composable.

Who gets to operate a building like this is the live question. Today, only a hyperscaler can, because a mixed fleet demands three scarce things at once: visibility into your own workloads, a second software stack, and capital. All three are getting cheaper. If the serving and storage layers keep abstracting the hardware, the tier below the hyperscalers inherits hyperscaler procurement economics, and the 2029 building gets built by companies that do not exist yet. If the abstraction stalls, the future is smaller: five private fleets, and a merchant market that still shops for the one best plane. Watching where those layers land will tell you more about 2028 silicon share than any chip roadmap.

What Would Make This Wrong

Three ways. NVIDIA absorbs the heterogeneity in-house, the way Rubin plus the Groq rack already gestures, and the fleet diversifies without the vendor list changing. Model architecture shrinks state faster than context grows it, latent attention’s 93% cache cut being the proof such compression is possible, and the old FLOPS logic claws territory back. Or the second software stack stays expensive enough that everyone below the top five rationally pays the CUDA premium forever. Each of these slows the fragmentation. None reverses it while decode stays memory-bound, and decode is where the tokens come from.

The Scoreboard

The framework compresses into a table: name the workload, find its binding constraint, and the winner falls out.

Three calls to hold me to. By end-2027, at least two of the five largest AI buyers serve one frontier model family across three silicon vendors in production. ASIC-based systems pass 35% of AI server shipments, with the gain coming from inference. And at least one non-hyperscaler inference provider runs a mixed NVIDIA and non-NVIDIA fleet at scale. If all three miss, the thesis was wrong and the one-plane world won.

Inference is becoming stateful. Weights, context, and KV cache accumulate in specific places, and moving that state increasingly costs more than choosing the theoretically fastest chip. So the compute comes to the data, rather than the data forever shuttling to the compute.

Data has gravity. Accelerators fall toward it.

• Constant-capability inference deflates roughly 10x per year: GPT-3-class went from $60 to $0.06 per million tokens in three years. GPT-4-class is down about 300x.

• The discount never applies to the frontier. Flagship pricing fell from $30 to $1.25, then rose back to about $5 with GPT-5.5. Enterprises buy the frontier anyway: open-source share of workloads fell despite far lower prices (Menlo Ventures).

• Paid demand out-multiplied the cuts. OpenAI went from $3.7B in 2024 to a reported $40B+ run rate; Anthropic from $1B to $65B in 19 months, with inference margin rising from about 38% to about 70% in a year (SemiAnalysis).

• Applications are the fastest revenue cohort in software history (Cursor: $100M to $3B in 17 months) at negative reported gross margins. Outcome pricing and model routing decide who keeps the windfall.

• Hyperscalers hold $1.7T of contracted backlog and just posted their first negative free-cash-flow quarters. The open question underneath: who owns GPU depreciation when cost per token falls 10x per generation.

• Nvidia manufactures the deflation and disclosed $1T+ of orders for it at a 74.9% gross margin. Power repriced the other way: PJM capacity went from $28.92 to $333.44 per MW-day in four auctions.

• Verdict: value migrates toward what cannot be deflated. Frontier capability and distribution at the top, watts and land at the bottom. The undifferentiated version of any business makes less.

This has happened before. Transistors, bandwidth, and electricity all deflated this fast, total spending never shrank, and who kept the money differed each time.

Run the same sorting on today's stack, and this is the answer the rest of the piece defends.

Direction of unit price, volume, and absolute profit pool by layer under one more 90% cost decline. Author’s assessment from the data below.

Thesis

Thesis. The price of a fixed unit of intelligence has fallen roughly 10x per year for four consecutive years. GPT-3-class output cost $60 per million tokens in November 2021 and $0.06 by late 2024, the 1,000x decline a16z documented as “LLMflation.” Every layer underneath that collapse posted record revenue in 2026 anyway.

The cheapest price to match a fixed benchmark score (a16z, Stanford HAI, Epoch AI). GPT-3-class fell from $60 to $0.06 per million tokens in three years; GPT-4-class is down ~300x since March 2023.

The August 2026 scoreboard: Nvidia has disclosed over $1 trillion of Blackwell and Rubin order visibility. SK hynix printed a 76% operating margin. Big Four capex is guided near $740 billion, up from $410 billion in 2025. Two labs whose prices fell all year run at a combined ~$90 billion of annualized revenue. And the next 90% decline is already scheduled: Rubin ships this half claiming another 10x cut in cost per token.

So the question is not whether inference gets 10x cheaper. It is which layers convert the deflation into profit, which absorb it as margin compression, and whether value migrates up the stack as infrastructure commoditizes, or whether consumption grows so fast that infrastructure captures enormous absolute dollars anyway.

The last piece in this series argued that value capture in AI infrastructure tracks scarcity, not spend. This one runs that framework through a moving target, and the short answer is that deflation does not move value up or down the stack so much as toward whatever cannot be deflated. At the top, that is frontier capability and distribution. At the bottom, watts. The chart below is the whole piece in one picture. The rest is the evidence.

The verdict by layer. Every claim in this picture gets its numbers in the sections below.

The Demand Response

Everything in this piece rests on one question: when the price falls 90%, does total spending rise or fall? Paid demand answered first. OpenAI grew from $3.7 billion of revenue in 2024 to a reported $40 billion-plus run rate by August 2026. Anthropic went from a $1 billion run rate to a company-stated $47 billion, and both cut constant-capability prices the whole way. Enterprise generative AI spend hit $37 billion in 2025, up 3.2x (Menlo Ventures). OpenRouter’s paid, revenue-weighted traffic went from 5 trillion to 25 trillion weekly tokens in six months. A 10x price cut met a demand response larger than 10x at every layer where money changes hands. A caveat on that sentence: it stacks different measures (revenue, spend, tokens, orders), not a controlled demand curve, so it shows direction rather than a measured elasticity. Still, every proxy points the same way, and none point the other.

Google’s disclosures put the ceiling on the volume story: 9.7 trillion monthly tokens in May 2024, 480 trillion a year later, 3.2 quadrillion by May 2026. That is 330x in two years. Two caveats: the series includes free surfaces like AI Overviews, and Tom Tunguz noted monthly additions decelerated in late 2025 before reaccelerating. Treat it as the shape of consumption, not its price tag; the paid numbers carry the argument on their own.

The demand response also survived its cleanest natural experiment. DeepSeek’s January 2025 efficiency shock erased $589 billion of Nvidia’s market cap in a day on the thesis that efficiency kills demand. Eighteen months later, Nvidia’s order book had doubled to $1 trillion. Satya Nadella’s same-day line, “Jevons paradox strikes again,” is now the consensus view.

Google-disclosed monthly tokens across its surfaces and APIs, including free and internal ones. May 2026 is 330x the May 2024 baseline.

The subtler force: the task itself is inflating underneath the price cut. OpenRouter’s 100-trillion-token study measured it on paid traffic. Between late 2023 and late 2025, the median prompt quadrupled from about 1,500 tokens to over 6,000, completions tripled, reasoning models went from negligible to more than half of all tokens, and programming, the most token-dense workload, grew from 11% of volume to over 50%. Stanford’s Digital Economy Lab separately measured agentic coding tasks at roughly 1,000x the tokens of a code-chat query.

Cheaper tokens do not just mean more queries. They mean each query is allowed to think longer. Inference-time scaling made tokens-per-task a design variable, and memory, networking, and power all inherit the consequence.

Paid-usage shifts from the OpenRouter study: prompts 4x longer, reasoning from near zero to a majority of tokens, programming from 11% to over half of volume.

Model Labs: Volume Wins, Margins Turning

The 10x discount applies to last year’s capability, never the frontier. OpenAI’s flagship launch price ran $30 (GPT-4), $10 (Turbo), $5 (4o), $1.25 (GPT-5). Then it turned back up, with GPT-5.5 tracking around $5 on third-party trackers. Ethan Ding made the canonical argument in mid-2025: frontier prices stay roughly flat, trailing prices collapse, and demand keeps migrating to the frontier. Menlo’s survey data confirms it. 66% of enterprise workloads upgrade to the newest model within months, only 11% switch vendors to save money, and open-source share fell from 19% to 13% despite far lower prices. Buyers pay for capability, not tokens.

That is the labs’ entire margin defense, and it has held revenue up. OpenAI went $3.7 billion to $13 billion to a reported $40 billion-plus run rate. Anthropic went $1 billion to $47 billion in seventeen months, company-stated and unaudited until the S-1s publish. Growing revenue an order of magnitude while cutting prices an order of magnitude is not a theory of elasticity. It is the income statement.

OpenAI flagship launch pricing against the cheapest GPT-4-class price from any provider. The discount applies to trailing capability; the frontier repriced upward in 2026.

OpenAI and Anthropic revenue trajectories, 2024-2026. Both labs cut constant-capability prices throughout this window.

Margins took longer. Reported figures, none company-published, put OpenAI around 40% gross in 2024 and 33% in 2025, missing its own 46% forecast. Anthropic ran -94% in 2024 and about 40% in 2025. Then the deflation reached the serving line: SemiAnalysis pegs Anthropic’s inference margin (revenue minus serving compute) at roughly 70% by mid-2026, up from 38% a year earlier, the same trajectory PitchBook’s compute-cost numbers trace ($0.71 per $1 of revenue in Q1, $0.56 projected in Q2). Inference margin is a narrower measure than blended gross margin; it excludes training amortization and free tiers. But the direction is the story: the serving business is compounding toward software economics on schedule with the hardware.

What remains unpaid is everything above the serving line: the frontier treadmill (each flagship costs more to train), reasoning-token inflation on the growth products, and free tiers at planetary scale. That is now the real race, a serving margin that improves with every hardware generation against training bills that scale with ambition. OpenAI’s internal 2030 target is reportedly 60%+ blended gross margin on ~$85 billion of inference spend; PitchBook’s bear case compresses Anthropic’s valuation 70-81% if blended margins stall below 35%. The mid-2026 serving data says the flywheel is turning. The S-1s will say how fast, and roughly $1.4 trillion of OpenAI data center commitments ride on the answer.

Reported lab margins against the 75-80% software benchmark. The 2026 Anthropic figure is inference margin (SemiAnalysis), serving compute only, not comparable to the blended figures. All press-reported.

Applications: The Biggest Beneficiary, Conditionally

A 90% price cut is a COGS windfall for the layer already growing faster than any software cohort in history. Cursor: $100 million to a reported $3 billion of annualized revenue in seventeen months. Lovable: $100 million within eight months of launch, $400 million by February 2026, with 146 employees. Claude Code: $1 billion roughly six months after general availability. Harvey $190 million, Glean $200 million, Sierra $100 million in 21 months. Enterprise application spend reached $19 billion in 2025, larger for the first time than the model spend beneath it.

The margins tell the other half. Cursor reportedly ran -23% gross for the quarter ended January 2026. Perplexity’s 2024 compute and model spend came to 164% of revenue. Replit swung between -14% and +36%, and Windsurf’s margin was described to TechCrunch as “very negative.” The cause is the treadmill: apps that compete on capability must serve the frontier, and the frontier never gets cheaper. The 10x discount goes only to products willing to run last year’s intelligence. In coding, Menlo pegs one vendor’s models at 42% of usage, so the category’s COGS is close to a single supplier’s price list.

Revenue ramps, January 2025 to June 2026. Cursor reached $2B annualized faster than any B2B software company on record.

Margin relief is a choice, not a default. Notion cut AI unit costs about 3x in two years by routing between commodity and frontier models per task. Cursor reportedly reached slight gross profitability in April 2026 on Composer, its in-house model serving the 80% of requests that do not need the frontier. Sierra prices per resolved issue, and Intercom’s Fin charges $0.99 per resolution, structures that charge for the outcome and pocket every future cost decline. a16z’s Sarah Wang now calls sky-high gross margins an “orange flag” for AI apps, a sign of too little AI in the product. That position only makes sense because the input curve falls 10x a year.

If inference falls another 10x, no layer gains more. Negative-23% at today’s prices is strongly positive at a tenth of them, provided tokens-per-task does not eat the decline, and outcome-priced agents convert deflation straight into gross profit. The apps that survive the next two years of negative margins inherit the best cost curve in software history. The ones pricing seats against frontier COGS will keep shipping their margin upstream.

Reported gross margins across AI-native applications against the SaaS benchmark. Every figure press-reported; none audited.

Hyperscalers: Volume Wins, Capex Is Forever

The July-August 2026 reports gave the buildout its best quarter yet. Azure grew 43% and crossed $100 billion in annual revenue; AWS accelerated to 37%, its fastest in eighteen quarters; Google Cloud grew 82%. The demand is contractual, not narrative: Microsoft RPO $678 billion (up 84%), AWS backlog $496 billion, Google Cloud backlog $514 billion. That is $1.7 trillion of signed, unrecognized demand, and all three said the same thing in the same season: demand exceeds supply. Jassy: AWS “will still not have enough capacity to meet all the demand” in 2026. Zuckerberg: Meta is fielding offers for its compute “at a significant premium.”

Cheaper inference is exactly what a hyperscaler wants. A cloud sells utilization, not tokens, and every cost decline widens the set of workloads that clear on its infrastructure. The math that squeezes the labs works for the cloud. Below it, the neocloud tier splits on the same test as every other layer. Bare GPU rental, meaning spot capacity with no software and debt against the depreciating asset, is where falling compute prices land first, and the prior piece scored it weakest in the stack. The version that earns its way out looks different: contracted demand with prepayments, disciplined hardware refresh, and optimization software that captures part of the efficiency it creates. A 10x-cheaper world sharpens that split. It does not sink the whole tier.

Combined Big Four capital expenditure, 2023-2026 guidance, with mid-2026 backlog disclosures.

The bill showed up one line lower in the same filings. Alphabet posted the first negative free-cash-flow quarter in its history (-$5.9 billion) while raising capex guidance twice, to $195-205 billion. Amazon’s trailing-twelve-month FCF swung from +$18.2 billion to -$7.6 billion. Meta reported quarterly FCF of $784 million, added $24.9 billion of new debt, and has suspended buybacks for two straight quarters. Microsoft, the healthiest, saw FCF fall 23%, and it extended its data center accounting life from 15 to 25 years, moving depreciation out of the near-term P&L without moving any cash. Amazon went the opposite way in 2025, shortening AI server life to five years. The two largest capex programs on earth are making opposite depreciation assumptions about the same asset, which tells you how unsettled the economic life of a GPU fleet is.

That disagreement is the question under the whole piece: when cost per token falls 10x per generation, someone owns the depreciation on the old generation, and where that risk sits is as investable a question as where the margin sits. Hyperscalers hold it on their own balance sheets, cushioned by software margins. Neoclouds hold it with debt collateralized by the depreciating asset itself. The labs mostly rent it. Nvidia manufactures it and books the replacement order. Every 10x cut in cost per token is also a 10x question about what the installed base is still worth.

The verdict: volume wins, margins normalize. A hyperscaler in 2020 was a software-margin business with a capex habit; in 2026 it is an industrial with a software attach. Cheaper inference raises the return on every installed watt and guarantees the watt count keeps compounding.

Most recent reported free cash flow against the year-earlier period. The demand story and the cash squeeze are the same story.

Semiconductors: Selling the Deflation

Semiconductors hold the strangest position in the stack. They are the source of the deflation, and they get paid more every year for manufacturing it. Roughly half the annual price decline is Nvidia’s roadmap executing on schedule. Hopper served a representative MoE model at about $0.20 per million tokens. Blackwell cut that to $0.05-$0.02 in production, with Baseten, Together, Fireworks, and DeepInfra all reporting up to 10x. Rubin, shipping this half, claims another 10x. Jensen Huang, on the May call: “Economics of AI of the future is tokens per dollar. Or dollars per token.” Nvidia sells the denominator.

The result is the least intuitive chart in this piece. The company causing the price collapse disclosed $500 billion of order visibility in October 2025 and “through 2027, at least $1 trillion” five months later, at a 74.9% GAAP gross margin. Deflation at 75 points of margin is not deflation for the vendor. It is the product.

Left: production cost per million tokens by hardware generation. Right: Nvidia’s disclosed order visibility doubled in five months.

One layer down, the foundry collects regardless of whose logo is on the die. TSMC’s June quarter: $40.2 billion of revenue, up 36%, at a record 67.7% gross margin, with CoWoS packaging sold out on 52-78 week lead times. Merchant GPU or custom ASIC, every accelerator in this piece routes through the same fabs. TSMC is the one company for whom the ASIC-versus-Nvidia fight is a rounding error.

The risk to the design layer is share, not demand. Custom ASIC shipments are growing 44.6% in 2026 against 16.1% for merchant GPUs (TrendForce), heading toward roughly 40% of AI servers by 2030, funded by the same four boardrooms that supply most of Nvidia’s revenue. Broadcom booked $30 billion-plus of AI orders in a single quarter and sees $100 billion in 2027; Trainium crossed a $25 billion run rate. Cheaper inference does not shrink the semiconductor pool; units, content per unit, and workload count all rise. It does redistribute the pool toward whoever owns the cost-per-token roadmap, and purpose-built inference silicon is closing that gap faster than it ever did in training.

Memory and Networking: The Complexity Tax

The memory layer taxes what the deflation unleashes. Million-token contexts, reasoning chains, and agents that re-read codebases are memory-bound before they are compute-bound. Context lives in HBM as KV cache and gets re-read at every decode step. The roadmap says so directly: H100 shipped 80GB at 3.35 TB/s in 2022; Rubin ships 288GB of HBM4 at 22 TB/s in 2026. Capacity per GPU rose 3.6x, bandwidth 6.6x, and memory is already about 45% of a Blackwell GPU’s build cost (Epoch AI).

The suppliers print the most extreme numbers in the stack. SK hynix: revenue up 257%, 76% operating margin. Micron: HBM sold out through 2026, gross margins in the 80s. Samsung’s CFO warns constraints get worse in 2027, with new fabs not helping before 2029 or so. Networking rides the same curve with a sturdier margin: Nvidia’s networking segment tripled to $15 billion in a quarter, Arista raised guidance to $12.6 billion with purchase commitments up nearly 3x, and Ethernet optics are on a path to a $100 billion market by 2030 (LightCounting). Arista’s CEO, on components: “the industry is going to have a two-year problem. I don’t think we get out of it as an industry till 2028.”

Memory per flagship data center GPU, 2022-2026. The workloads cheap inference enables are memory-bound before they are compute-bound.

The prior piece scored HBM as a cyclical shortage wearing a structural costume, and nothing here revises that. What a 10x decline changes is timing: every added order of magnitude of token demand extends the shortage, and the workload mix is shifting toward exactly the shapes that maximize memory content per unit of compute. Memory stays a trade, not a holding. It is currently the best trade in the stack, set against the strongest base rate in semiconductors: three well-capitalized suppliers adding capacity into 70%+ margins has always, eventually, mean-reverted. HBM has real reasons to run longer than the usual DRAM clock. Packaging qualification, CoWoS coupling, and multi-year hyperscaler contracts all slow the supply response. Longer is not never. Networking is the durable half of the pair: Arista’s 62-64% gross margins sit on software moats and switching costs the memory names do not have, which is why the prior piece scored it overweight while scoring HBM a trade.

Data Centers and Power: The Layer That Never Deflates

Here the pattern inverts. Everything above this layer gets cheaper per unit every year, and watts do not. PJM, the grid operator for the largest data center corridor on earth, cleared capacity at $28.92 per megawatt-day for 2024/25, then $269.92, $329.17, and $333.44, a price that would have hit roughly $530 without the federal cap. The market monitor attributes 46% of the last four auctions’ $63.6 billion cost to data centers (PJM disputes the framing in part). Northern Virginia vacancy is 0.3% with rents roughly doubled since 2022. GE Vernova’s turbine backlog and reservations: 116 GW, sold out into 2031. The scarce input under a 10x-cheaper token is not silicon. It is the interconnection queue, where the median project waits more than five years.

Google supplied both sides of the efficiency argument at this layer. Energy per median Gemini prompt fell 33x in twelve months, to 0.24 Wh, a real engineering feat. Total Google electricity rose 37% in 2025 anyway, to about 3.5x its 2019 level, with the largest emissions increase the company has ever reported. Per-unit efficiency times volume growth equals a bigger bill. It is the same arithmetic Jevons ran on British coal in 1865, now with quarterly disclosure.

PJM capacity clearing prices by delivery year. Compute deflated 10x over the same window; grid capacity repriced 11x the other way.

This is the cleanest structural read in the piece: power is the only layer where unit prices are rising, across capacity auctions, colocation rents, turbine lead times, and PPA strikes, while everything above it deflates. The precise version of the claim: scarce capacity in the markets where AI demand concentrates has repriced dramatically, while less-stressed grids have not, a limit What Breaks the Thesis takes up. Goldman has U.S. data center power demand going from 31 GW in 2025 to 66 GW in 2027. The IEA has global data center electricity nearly doubling to 945 TWh by 2030. Hyperscalers have signed roughly 10 GW of nuclear PPAs stretching into the 2040s. Cheaper tokens make all of it bigger, because the binding constraint on token supply is now megawatts. The owners of scarce megawatts are the only participants in this stack with pricing power they did not have to engineer.

Google’s own disclosures: per-prompt energy down 33x, tokens up 6.7x year over year, total electricity up 37% in 2025.

What Happened the Last Three Times

This is not the first technology to deflate 10x a year into exploding demand, and history is unanimous on the headline: unit deflation has never shrunk the aggregate. Transistor prices fell roughly a billion-fold from 1971 to 2022; industry revenue grew from about $1 billion to $791.7 billion. Real electricity prices fell about 48x across the twentieth century; consumption rose about 630x. IP transit fell from roughly $1,200 per Mbps to under $1, and traffic doubled annually straight through the telecom bust.

The variance worth studying is who kept the money, because the three cases split three ways. In semiconductors, the deflation manufacturers kept it (Intel, then TSMC and Nvidia) because making the next cheap unit was itself the monopoly. In bandwidth, the infrastructure owners kept nothing. WorldCom and Global Crossing went bankrupt on correct demand forecasts, and value moved up to the applications built on cheap bits: Google, Netflix, AWS. In electricity, no one kept outsized margins for long; regulators converted utilities into rate-of-return businesses and the surplus flowed to everything electrified. Deflation always sorts the layers. Which layer wins depends on where the hardest-to-copy scarcity sits.

Three technologies whose unit prices collapsed. The aggregate never shrank; the winning layer differed each time.

So Does Value Migrate Up the Stack?

Map those three endings onto this stack and the headline question stops being binary. The bandwidth ending, where infrastructure commoditizes and applications capture, applies selectively: it is the likely fate of undifferentiated GPU rental and API resale, which is why the neocloud tier already trades the way it does. The semiconductor ending applies where making the next unit of deflation is itself the moat: Nvidia’s roadmap, TSMC’s process lead, the HBM oligopoly while the shortage holds. The electricity ending is roughly where hyperscalers converge: steady, capacity-constrained, utility-like returns. The difference is that nobody regulates PJM prices down to a rate base, which is why power keeps the only rising unit prices in the stack.

Value migrates toward both ends of the stack, and within every layer it migrates away from the undifferentiated version of the business. Up, toward applications that own customers and price outcomes; a 90% COGS decline lands there as gross profit, if tokens-per-task and frontier dependence do not eat it first. Down, past the deflating compute layer, toward watts, land, and interconnection queues no roadmap can 10x. The compression is not a place in the stack; it is a business model. Token resale, GPU rental without software, and thin wrappers on someone else’s frontier model make less wherever they sit. An outcome-priced app, a moated switch vendor, and a scarce-megawatt owner make more in the same cycle. The prior piece’s framework survives with one amendment: value capture tracks how replicable a business is; deflation is the speed at which replicability gets tested.

Positioning. As a stance: own the deflation manufacturers (Nvidia, TSMC) and the un-deflatable bottom (power, grid equipment, and scarce-megawatt owners: the Vertiv and GE Vernova layer). Both ends hold pricing power the middle cannot reach. Treat memory as a trade while the shortage holds, since the capacity response is funded and dated, and hold the networking half of that pair, where the moats are software. Underwrite applications on pricing model and routing discipline, not growth rate. Underweight anything whose only product is someone else’s tokens at a markup: bare GPU rental, API resale, and seat-priced wrappers pinned to frontier COGS. A neocloud exits that bucket with contracted demand and software on top of the compute, not with more GPUs. The labs are venture bets on a terminal margin no one has demonstrated. Size them like it.

What Breaks the Thesis

Elasticity below 1 is the honest bear case. Every dollar figure here rests on demand multiplying faster than prices divide. The early counter-evidence: Google’s monthly token additions decelerated 57% between mid and late 2025 (Tunguz), enterprise budgets are moving from experimentation into permanent IT lines where CFOs enforce ROI, and agents are getting more efficient. Anthropic claims up to 65% lower token use on comparable tasks for its newest models. If tokens-per-task stops inflating while prices keep falling, aggregate inference revenue can fall even as usage rises, and $1.7 trillion of backlog converts slower than the depreciation schedules assume. DeepSeek tested this for one quarter. A real efficiency-led shortfall would test it for eight.

The numbers in the middle of this piece are the softest. Lab margins, Cursor’s -23%, Perplexity’s 164%: all reported by The Information or PitchBook, none audited, and both lab run rates are self-stated at funding events. The S-1s on file will replace several of these figures within months; this piece gets updated when they publish. Materially better audited margins would confirm the serving-line turn. Materially worse, and the application layer’s supplier is shakier than it looks. The power argument has the same shape: it leans on PJM, the most stressed grid in the country. Other regions have not repriced 11x, and a fast turbine-and-interconnection response would soften the claim by 2028.

Vendor-claimed hardware economics deserve their asterisks. The 10x-per-generation figure is Nvidia’s number for MoE inference under NVFP4, seconded by clouds with marketing incentives, and Rubin’s claimed 10x is a spec sheet until it ships at fleet scale. If the real generational gain is 3x, the deflation slows and the demand response weakens with it. That would, perversely, strengthen the memory and power arguments while weakening everything else here.

Summary

Inference getting 10x cheaper is the mechanism of this market, not a threat to it. The deflation is manufactured at the silicon layer and sold at 75% gross margin. It passes through hyperscalers, who convert it into $1.7 trillion of contracted backlog and permanently higher capital intensity. It reaches the labs as a treadmill that grew revenue an order of magnitude while blended margins lagged the software benchmark, though serving margins reportedly hit 70% by mid-2026, the first hard evidence the deflation is reaching the labs’ own P&L. And it lands at the application layer as the best input cost curve in software history, capturable by whoever prices outcomes instead of reselling tokens. Underneath it all, the one layer with rising unit prices, power and land and grid capacity, holds the pricing power everyone else has to engineer. Demand has out-multiplied the price decline at every test for four years. History says unit deflation never shrinks the aggregate; it sorts by scarcity. This cycle is sorting the same way, and in every layer the undifferentiated version of the business will make less.

The Fourth Wave

New engineering disciplines get minted the same way every time. A layer of the stack becomes economically decisive, a job title crystallizes around it, salaries run ahead of the talent supply, and a tooling industry forms underneath. Data science ran the arc first: coined around 2008 at Facebook and LinkedIn, anointed by Harvard Business Review in 2012, postings up 256% from 2013 to 2018. Data engineering followed once pipelines mattered more than notebooks — the fastest-growing tech job of 2019 at +50% year over year (Dice), with Snowflake’s record IPO and Databricks, $134B at its late-2025 round, as the tooling wave behind it. ML engineering ran it third: Indeed’s #1 US job in 2019, postings up 344% in three years.

Four waves, one pattern: term emerges, anointing moment, posting boom, tooling industry

The fourth is forming now, and the job is easy to state: take a trained model as a fixed artifact and maximize useful tokens per second, per GPU, per dollar, per joule. Anthropic’s posting for a “Performance Engineer, Inference Systems” describes it as “sizing the gap between actual fleet performance and theoretical rooflines.” The title hasn’t settled — inference engineer, performance engineer, model serving engineer — and LinkedIn doesn’t track it as a category yet, which is roughly where “data scientist” stood in 2010. Supply is thin for a structural reason. The work sits at the intersection of GPU microarchitecture, distributed systems, and numerical methods, a combination no university program or bootcamp produces. Most of the people who can do it trained themselves inside a handful of labs, chip companies, and open-source serving projects.

Tokens Went Quadrillion

Start with demand, which has outrun every forecast. Google processed 9.7 trillion tokens a month in May 2024, 480 trillion a year later, and over 3.2 quadrillion by May 2026 — call it 330x in two years. Microsoft ran 100 trillion tokens through a single quarter of FY25, up 5x. OpenRouter, which routes across 400+ models, went from 5 to 25 trillion tokens a week in six months. What changed is that tokens stopped being a chat phenomenon: reasoning models spend thousands of tokens thinking before they answer, and agents reread their entire transcript on every step. Jensen Huang’s claim in early 2025 that reasoning AI would need “100 times more” compute sounded like salesmanship. The token disclosures since then have made it look like arithmetic.

Each gridline is 10x. When demand compounds like this, the marginal cost of a token becomes the most important line in the P&L

The Price of Intelligence Falls 10x a Year

While demand grew 330x, the price of any fixed level of capability collapsed. GPT-3-class tokens cost $60 per million in late 2021 and $0.06 three years later — 1,000x, or 10x a year (a16z). Epoch AI, measuring the price to hit fixed benchmark scores, found declines between 9x and 900x per year depending on the task, with the caveat that the steepest declines are recent and may not persist. Sam Altman stated it as an operating assumption: the cost to use a given level of AI falls about 10x every 12 months.

Where does the deflation come from? Not primarily hardware — 2x per generation doesn’t compound to 10x a year. The balance comes from the serving stack: batching, memory management, quantization, caching, speculation, the techniques in the back half of this piece. The cleanest tell is on Alphabet’s earnings calls, where the company said it cut Gemini’s serving cost 78% in a single year while capex rose. Someone engineered that number, and this piece is about the people who do.

Constant capability, collapsing price. Hardware accounts for a fraction of the slope; serving software does the rest

Inference Is Now Most of AI Compute

The workload mix flipped underneath the buildout. Deloitte puts inference at about two-thirds of all AI compute in 2026, up from one-third in 2023. Training is a periodic capital project; inference is a permanent operating cost that grows with usage, and it is what the capex increasingly buys — roughly $500B of hyperscaler spend in 2025, with 2026 guidance totaling $660–690B. The inference market itself is sized at $106B in 2025, heading for $255B by 2030 (MarketsandMarkets). This is what gives serving teams their unusual leverage. Against $100B of deployed infrastructure, a 15% fleet-wide throughput improvement frees about $15B of effective capacity. Few engineering roles have ever had a larger denominator under them.

Inference went from a third of AI compute to two-thirds in three years, on a capex base approaching $700B a year

What the Market Pays

Anthropic’s posting for an inference performance engineer carries a salary range of $350,000–$850,000. OpenAI’s equivalent posts at $295,000–$555,000, and levels.fyi puts senior IC total compensation at both labs between $600K and $1.15M+ with equity. The reported tail runs much further: Altman said Meta offered his staff “$100 million signing bonuses”; the WSJ reported Meta’s package for Andrew Tulloch — a performance engineer, not a research celebrity — at up to $1.5B over six years (Meta disputed the details); and in December 2025 NVIDIA paid a reported ~$20B to license Groq’s low-latency inference technology and hire its leadership, three months after Groq had raised at $6.9B. For calibration, the average San Francisco data scientist earned $167K at the peak of that wave.

Posted ranges, reported totals, and the reported tail — reported figures flagged as such throughout

Counting the Field

Prices like those imply scarcity, so it is worth asking directly: how many of these engineers actually exist? No labor statistic tracks the title, so I counted. On August 9, 2026 I pulled every public careers board I could parse — the ATS endpoints behind the major labs, serving companies, and infrastructure players — and tallied open roles whose title or core description is inference-serving performance: inference engineer, model serving, inference runtime, GPU kernel work, performance engineering in a serving context. Research scientists, hardware ops, and go-to-market roles were excluded. Seventeen boards were countable. The total came to roughly 58 open roles, and 24 of them sit at two companies, OpenAI and Anthropic.

That number is a floor. NVIDIA, AMD, Meta, and Mistral run JavaScript-only portals that defeat a hand count, and NVIDIA’s own search surfaces many live TensorRT-LLM and inference-performance reqs. But even doubled or tripled, the shape holds: the visible open market for the discipline that manufactures AI’s cost curve is measured in dozens of seats, set against 2026 capex guidance of $660–690B. Two details in the data say more than the total. Fireworks — a $17.5B serving company — lists zero dedicated inference-performance openings; at the top of this market, roles fill through networks before they ever reach a board. And there is no standard title: seventeen boards produced inference engineer, performance engineer, kernel engineer, model serving, inference runtime, and inference systems, with no two companies using the same one. Postings for “data scientist” looked exactly like this in 2011, the year before HBR named the job.

Data Gravity hand count, August 9, 2026. Floors marked ≥ where boards truncate; NVIDIA, AMD, Meta, and Mistral could not be counted

The Constraint: Memory, Not Compute

To understand what these ~58 postings are paying for, one piece of physics is enough. Generating a token in the decode phase means streaming essentially all of the model’s active weights from HBM to the compute units — for one token. That is about two floating-point operations per byte fetched, against an H100 that does ~990 dense FP16 TFLOP/s on 3.35 TB/s of bandwidth: the chip needs ~295 FLOPs per byte to stay busy. Decoding one request at a time uses a low single-digit percentage of the machine while the compute units wait on memory. And the imbalance is widening by design — peak FLOP/s has grown 3.0x every two years industry-wide against 1.6x for DRAM bandwidth (Gholami et al.). V100 to B200: compute up ~18x, bandwidth up ~8.6x.

Every technique below attacks this constraint in one of two ways: move fewer bytes per token, or spread each byte across more tokens.

Compute keeps outgrowing the memory that feeds it. Decode is bandwidth-bound, so unoptimized inference wastes most of the silicon

1. Continuous Batching

The highest-leverage idea in serving is scheduling. If one request can’t saturate the GPU, run many at once, so each pass through the weights produces tokens for dozens of users. The catch is that naive batching holds every request until the slowest one finishes. Orca (OSDI ‘22) fixed this with iteration-level scheduling — requests join and leave the batch between individual decode steps — and reported 36.9x throughput against NVIDIA’s FasterTransformer at the same latency. Anyscale’s public benchmark on OPT-13B decomposed the gain: continuous batching alone was worth ~8x over naive static batching, and adding vLLM’s memory management took it to 23x. Every serious serving stack is now built around iteration-level scheduling; it is the largest single efficiency lever in production inference.

One scheduling idea, 23x the throughput from the same silicon

2. PagedAttention

Big batches created a memory problem. Every in-flight request carries a KV cache — the attention keys and values of its context — which grows token by token and can reach gigabytes per sequence. Servers used to allocate it in contiguous slabs sized for the maximum possible length, and the vLLM team measured the result: 60–80% of KV memory wasted on fragmentation and over-reservation. Their fix, PagedAttention (SOSP 2023), is the oldest idea in operating systems pointed at attention: allocate the cache in small pages, on demand, shared across requests. Waste fell below 4%, the recovered memory became batch size, and throughput rose 2–4x over the prior state of the art — up to 24x over HuggingFace Transformers. LMSYS’s Chatbot Arena served the same traffic on half the GPUs. A 40-year-old OS idea, applied in the right place, halved a production service’s compute bill.

Under 4% waste means larger batches from the same HBM — the source of the throughput gains below

3. Quantization

If decode speed is set by bytes moved, the bluntest lever is smaller bytes. Weights went from 16 bits to 8 to 4 in about three years, and the surprise is how little accuracy it costs when done carefully. GPTQ showed one-shot 4-bit quantization at 175B scale back in 2022; AWQ refined which weights need protecting. By 2025 it wasn’t an aftermarket modification anymore — OpenAI shipped gpt-oss-120b natively 4-bit, putting a 117B-parameter model on one 80 GB GPU, and Blackwell added a hardware 4-bit format. NVIDIA’s own conversion of DeepSeek-R1 to it measured ≤1% accuracy degradation. Each halving of bits roughly doubles effective memory bandwidth and halves the GPUs required; a16z credits quantization with at least 4x of the historical cost decline. The skill is knowing, and proving with evals, where 4 bits is enough.

FP16 to FP4 takes a 70B model from multi-GPU to one card with headroom, at ~1% measured accuracy cost

4. Prefix Caching

Production traffic repeats itself constantly — system prompts, retrieved documents, conversation history — and agents resend their whole transcript on every step. Prefix caching stores the KV cache of shared prefixes and reuses it, turning repeat prefill compute into a lookup; SGLang’s RadixAttention automates the reuse and gets up to 6.4x throughput on multi-call workloads. What makes this technique interesting is that it escaped the infrastructure layer and became pricing. Cached input tokens sell at 10% of the base price at Anthropic and DeepSeek, 50% at OpenAI. When an optimization gets its own line on a public price sheet it has become market structure, and application developers now engineer their prompts around it. Where that cached state physically lives — and what it is doing to the NAND market — was the subject of the storage piece.

A systems optimization, surfaced as an exchange rate: a cached token costs 10–50% of a fresh one

5. Disaggregated Serving

Prefill and decode are opposite workloads sharing a GPU. Prefill is compute-bound and parallel; decode is memory-bound and latency-sensitive; a long prefill stalls every decode stream on the card. Disaggregation splits them onto separate GPU pools, ships the KV cache between them, and lets each pool scale against its own latency target. DistServe (OSDI ‘24) measured 7.4x more requests served within latency limits, and gave the field its scoreboard metric — goodput, throughput that actually meets its latency promises. Moonshot runs this architecture in production for Kimi at +75% requests on the same hardware. NVIDIA productized the pattern as Dynamo, now its flagship serving layer. From academic paper to NVIDIA’s headline software product took about two years; this field industrializes its research nearly as fast as it publishes it.

Two pools, each tuned to its own physics — and a new metric, goodput, that only counts throughput when latency promises hold

The Rest of the Toolkit

Four more levers compound on top of these five, each a variation on the same two moves. Architectural attention changes shrink the KV cache before it exists: grouped-query attention, the Llama default, cuts it 8x, and DeepSeek’s multi-head latent attention compresses it 93.3% while raising maximum throughput 5.76x — an under-appreciated input to how DeepSeek priced its API. Speculative decoding turns cheap draft tokens into verified ones with provably unchanged outputs; three research generations took it from 2–3x to 6.5x and made it hold up at production batch sizes. Kernel engineering extracts more from silicon that already exists — FlashAttention-3 lifted attention from 35% to 75% of H100 peak with identical math, and PyTorch’s gpt-fast reached 9.6x on an unmodified A100 by stacking compilation, quantization, and speculation. And mixture-of-experts moved frontier models from activating 27.6% of their weights per token to ~4–5% in two years, while distillation put o1-mini-beating reasoning in a 32B model. Every one of these went from paper to production default in under three years. The field is closer to its beginning than its middle.

The compounding levers beyond the core five — all recent, all still improving

The Tooling Wave

In every prior wave the practitioners got expensive first and the tools that leverage them got funded second. This time both are happening at once. Inside roughly eighteen months: Fireworks AI at $17.5B on more than $1B of annualized revenue; Baseten reported at $13B, five months after pricing at $5B; Together AI at $8.3B on $1.15B of bookings; Cerebras at a ~$66B first-day market cap; Modal at $4.65B; OpenRouter at $1.3B. Even the open-source engines became companies — vLLM’s creators raised a $150M seed at $800M as Inferact, and the SGLang team spun out at a reported ~$400M — while the frameworks themselves consolidated into infrastructure: vLLM runs Amazon Rufus and LinkedIn, SGLang is xAI’s default engine, HuggingFace’s TGI went into maintenance mode, and NVIDIA elevated Dynamo to a first-class product alongside CUDA.

Why the category attracts this capital comes down to unit economics. DeepSeek published a theoretical 545% cost-profit ratio on its serving fleet — actual margins are lower, since the figure assumes all traffic billed at full R1 pricing, but the disclosure showed what engineered serving efficiency does to a P&L. Alphabet’s 78% Gemini cost reduction is the same lever at hyperscale. In this cycle, serving efficiency is not a cost center. It is where AI gross margin comes from.

The tooling wave, funded in real time — before the discipline powering it has a standardized job title

What Breaks the Thesis

Three real risks, in descending order. First, commoditization from above. Dynamo, vLLM, and SGLang are absorbing the median case the way dbt absorbed the median data-engineering task, and MLOps is the cautionary precedent — the #1 job of 2019 whose middle was largely absorbed by SageMaker and Vertex within five years. If the defaults get good enough, the middle of the skill distribution gets cheap even while the top stays scarce. The difference worth watching: optimization value here scales with fleet size, and the fleets are growing 70% a year, which MLOps never had underneath it. Second, hardware relief. If HBM bandwidth scales faster than expected, or memory-centric architectures gain real ground, the memory wall softens and some of the software leverage compresses with it. The trend line points the other way so far, but it is a bet on physics roadmaps, not a law. Third, the field’s own numbers run hot. NVIDIA’s “30x” for Blackwell inference compares a rack-scale FP4 system against an air-cooled FP8 Hopper baseline; independent analysis puts chip-level gains nearer 4x, with the rest unlocked by software and system design. The gains in this piece are real and compounding, but vendor marketing routinely overstates them — one more reason the people who can measure honestly are worth what they cost.

Summary

The center of gravity in AI economics has moved from training models to serving them. Inference is two-thirds of AI compute on a capex base heading toward $700B a year, and the price of constant-capability tokens falls 10x annually — a decline manufactured mostly in software, one technique at a time. The techniques are young and still steepening: batching went to 23x, speculative decoding from 2x to 6.5x in three years, FlashAttention from 35% to 75% of peak, active parameters from 28% to 4%, and each moved from paper to production default in under three years. Against that stands the supply side this piece measured directly: roughly 58 open roles across seventeen countable careers boards, two dozen of them at OpenAI and Anthropic, with no standard job title among them. That ratio — a few dozen visible seats leveraged against hundreds of billions in deployed capital — is the whole investment case in one line. The value sits in three places: the practitioners, who are being priced like the scarce input they are; the platforms that leverage them across thousands of customers, which is where the prior waves’ Snowflakes and Databrickses came from; and the serving margin itself, which now determines which AI applications survive their own growth. The watch list from here: whether the postings census grows into a boom the way data science’s did after 2012, where the vLLM and SGLang contributors land, how far cached-token discounts deepen, and which providers can show — as Google did with 78% — that they manufacture their own margin. Squeezing more intelligence out of the same silicon is now the central economic activity of the AI buildout, and it is being done by hand, by a group of people you could fit in one lecture hall.

Data Gravity covers AI infrastructure, compute economics, and durable software systems. Related coverage: Why KV Cache and Memory Drive AI Economics and Why AI Is Becoming a Storage Workload.

Thesis. The most expensive processor in the data center spends most of its time waiting for bytes. Yet the consensus AI-infrastructure stack has three headline layers — GPUs, networking, HBM — and treats storage as plumbing, because of one assumption: training is a storage workload, but inference is stateless. Tokens in, tokens out, nothing to keep. That assumption is now wrong. Stated precisely: inference is becoming a state-management problem — compute increasingly exists to transform state, and storage is where state persists. The clearest evidence is on the API price sheets: OpenAI, Anthropic, and Google sell cached input tokens at ~90% off; DeepSeek, which parks cached context on SSDs, at ~99% off. Those discounts exist because a stored byte of context replaces recomputation — every gigabyte of remembered context is compute you never have to buy. Long context, agents, RAG, and multimodal serving are manufacturing that state at extraordinary rates, and it lands on flash — which is why NAND, a $270B market growing ~281% year-over-year, is repricing faster than any memory cycle on record. The strategically interesting line in AI infrastructure is no longer just the network between GPUs. It is the boundary between memory and storage.

AI storage by the numbers: NAND market size, KV cache per session, cached-token discounts, record eSSD quarter, HDD lead times, Google token volume

The Bottleneck Moved

Start with a fact that should be better known: during generation, a GPU is mostly not computing. Emitting one token means streaming every active model weight and the session’s entire KV cache out of memory — well over 100GB per step for a 70B-class model — and the arithmetic waits on the bytes. The imbalance is quantifiable: at batch size one, decode performs roughly 2 FLOPs for every byte it reads, while a modern accelerator is provisioned at ~300–600 FLOPs per byte of memory bandwidth — the math units are structurally >99% idle unless batching fills them, and batch size is capped by how much KV cache fits in memory. So memory capacity, not FLOPs, increasingly sets cost per token. The weights, meanwhile, are the most-moved bytes in the data center: consumed at full HBM bandwidth, a 140GB model is read end-to-end about 24 times per second — on the order of two million complete passes a day per serving instance. Penguin Solutions summarizes the resulting split: serving is roughly 30% compute, 70% memory. AI infrastructure has four layers — compute, memory, network, storage — and the last three are all versions of the same problem: how fast can you move bytes to the arithmetic? Training answered with HBM and NVLink, and the industry concluded the answer was permanent. But inference moves different bytes — context, weights, adapters, embeddings — and, critically, its bytes outlive the request. State that survives has to land somewhere, and each tier it can land on trades latency for capacity and cost. That is why inference optimization is turning into memory-hierarchy design rather than FLOPs procurement — and why the bottom of the hierarchy, storage, has stopped being a passive archive and become an active participant in serving.

The Blind Spot

Follow the dollars, not the discourse. Nearly all AI-memory coverage is about HBM — a market that even bullish forecasts sized around $55B for 2026, before this year’s repricing. NAND flash, which almost nobody covers, is a $270.6B market in 2026, heading for a projected $379B in 2027 (TrendForce, May 2026). For scale: the entire hard-drive industry books less revenue in a year (~$23B) than the top five NAND vendors now book in a single quarter ($38.9B). Storage stayed out of the AI narrative because the serving story said it could: if inference keeps nothing, storage is just where cold data goes to be cheap. The last eighteen months quietly falsified that premise.

The dollars vs the discourse: 2026 market size for DRAM, NAND, HBM and HDD

Training Was Always a Storage Workload

The training side was never light on storage — it was just a batch problem. MLPerf’s checkpoint benchmark puts numbers on it: a 70B-parameter model writes a 912GB checkpoint; a 1T model, 15TB. Cadence scales with failure rate — Meta’s Llama 3 run logged 419 interruptions in 54 days on 16,000 GPUs, and at 100,000 accelerators the math implies a checkpoint every ~90 seconds, written in under five seconds: a ~3.6TB/s burst. DeepSeek open-sourced the machinery it built for exactly this — 3FS, a flash-native filesystem sustaining 6.6TiB/s of aggregate read. But checkpoints are periodic, sequential, and predictable: a solved class of problem. What changed in 2025–26 is the other side. Serving stopped being stateless.

Training storage demands: checkpoint size by model scale, from 105GB at 8B parameters to 15TB at 1T

Inference Was Supposed to Be Stateless

The state that broke the assumption is the KV cache. As a model reads your context, it builds a scratchpad of intermediate results — attention keys and values for every token — so that each new word only has to consult the notes, not reread the raw text. Throw the scratchpad away and the model must reprocess the entire conversation from scratch before it can say anything. That scratchpad is big: for a 70B-class model it runs ~0.33MB per token, so a 128K-token session carries ~42GB of state — on top of ~140GB of model weights. Four concurrent long sessions exceed the HBM of any GPU ever shipped. Concurrency multiplies it without limit: VAST Data’s arithmetic for a consumer-scale service puts 100,000 concurrent users at ~45PB of retained context — three orders of magnitude beyond what HBM and DRAM can hold. Solidigm’s framing is the right one: an inference service becomes memory-limited long before it becomes compute-limited.

And the KV cache is only the loudest entry in a growing inventory. Model weights are now hot storage objects — DeepSeek-V3 is 688GB on disk, and serving fleets shuffle them constantly. LoRA adapters, a few hundred megabytes each, get swapped per-request by the thousands. Agent memory persists between sessions by design. Each item is small next to a checkpoint; together, multiplied by users, they are the largest new data category in the data center.

The state inventory: what inference now has to keep, from 200MB LoRA adapters to a 45PB context pool, on a log scale

Agents Make The Storage Problem Worse

Agents converted context from a per-request cost into a durable asset. The best public trace data comes from LMCache’s replay of 739 real Claude Code sessions: on average, a session’s context grows from ~20K to ~115K input tokens over an hour-long conversation — a 5.7x inflation, re-sent on every turn, of which 97% is reusable prefix. At 70B-class density, that one session is ~7GB of state at the start and ~38GB by the end. Now scale it: reasoning-and-tool-use models went from negligible to more than half of all tokens on OpenRouter during 2025, and Kioxia now calls agentic AI “the primary growth driver for NAND demand expansion.” A session that lives for an hour, pauses, and resumes tomorrow is not a cache-in-DRAM problem. It is a storage problem.

Agentic sessions inflate context: median Claude Code session grows from 20K to 115K input tokens in one hour

The Quieter State: Generated Media

Multimodal generation is compounding storage demand from the output side. Every generated video, image, and audio clip becomes a persistent asset the moment it exists — and Western Digital’s case work found each AI-generated video is stored at least seven times across platforms: the creation site, the creator’s copies, and every social network it’s posted to. WD’s summary is the structural point of this whole piece: storage demand is cumulative and persistent, unlike compute demand. That asymmetry is underpriced — compute demand can drop overnight on a single efficiency breakthrough; stored bytes never un-accumulate. A GPU finishes its job and moves on. The bytes stay.

The Market Already Prices State

Here is the mechanism in one sentence: reusing context the model has already seen normally means paying full compute to reread it — unless the scratchpad was saved, in which case you pay a disk read. The price sheets quantify the difference: cached input runs ~10% of fresh across OpenAI, Anthropic, and Google, and ~1% at DeepSeek, whose cache has lived on disk since 2024. The vendor benchmarks — all self-run, so treat the exact magnitudes loosely — point the same direction — WEKA reports 20x faster time-to-first-token at 128K context from flash-resident cache; Mooncake, the architecture behind Kimi, reported 75% more requests served on the same GPUs by pooling cache across DRAM and SSD. Storing a long prefix costs pennies per month; recomputing it costs the same dollars every single time. Once context is reused — and agents guarantee reuse — flash wins the arithmetic.

Cached vs fresh input token pricing across OpenAI, Anthropic, Google and DeepSeek — the market prices stored state at 90-99% off recompute

The Arithmetic of Remembering

Make it concrete. A 50,000-token prefix — a system prompt plus a codebase, or an agent’s accumulated working context — carries ~16.5GB of KV state on a 70B-class model. Rereading it costs ~$0.25 at prevailing fresh-input prices, every single time. Storing it costs ~$1.32 a month at ordinary cloud block-storage rates, and serving it back costs ~$0.025 per reuse at cached rates. The breakeven is six reuses a month. An agent reuses its context on every turn — it clears that bar within its first few minutes. At a thousand reuses a month, the choice is $250 of recompute against $26 total, of which the storage itself is $1.32. That asymmetry is why the discounts exist, why every serving stack now tiers to flash, and why “stateless” is ending: remembering costs an order of magnitude less than rethinking.

The arithmetic of remembering: monthly cost of recomputing a 50K-token prefix vs storing its KV cache on flash — breakeven at ~6 reuses per month

Where the State Lands

So does storage actually enter the inference hot path? Yes — through one door. An LLM answers in two phases: prefill (ingest the whole context — compute-heavy) and decode (emit tokens one at a time — memory-bandwidth-heavy). Decode stays welded to HBM and is not leaving. Prefill is the door, because a persistent KV cache makes most of it unnecessary — and NVIDIA has now formalized where that cache lives. Its Dynamo serving stack names the tiers like memory levels: G1 (HBM) → G2 (DRAM) → G3 (local SSD) → G4 (networked storage). At CES 2026 it went further, announcing BlueField-4-based “inference context memory” systems — a pod-level flash tier with a claimed 5x tokens per second, launched with a partner list that reads like the entire storage industry: DDN, Dell, HPE, IBM, Pure, VAST, WEKA and five others. The silicon is following: Kioxia’s GP1 series — built on its low-latency XL-FLASH media specifically for KV cache — samples this year at more than 10 million random-read IOPS, and Samsung’s PCIe Gen6 drive loads a 40GB model in 1.4 seconds. NVIDIA’s storage-certification program now defines a dedicated KV-cache I/O profile. When the GPU vendor starts speccing drive workloads, storage has stopped being plumbing.

Two phases, one door: decode stays welded to HBM; prefill is where a stored KV cache replaces recomputation, reloading through Dynamo’s G1-G4 hierarchy

Related: Why Prefill and Decode are Splitting the GPU

The NAND Shock

The commodity market noticed before the narrative did. NAND contract prices rose ~55–60% quarter-over-quarter in Q1 2026 and another 70–75% in Q2 — records, against the low single digits of a normal cycle — and the top five enterprise-SSD vendors booked a record $18.5B quarter, up 86% sequentially, which TrendForce attributes explicitly to AI-agent services. The supply side cannot chase it: 2026 industry capex puts $61.3B into DRAM against just $22.2B into NAND, mostly process migration rather than new wafers. So buyers are locking the future instead — SanDisk holds $42B in contracted revenue, and Kioxia has roughly half of its 2028 output already committed. TrendForce doesn’t expect the tightness to clear before late 2027.

The NAND shock: record quarterly contract price increases through 2026, and top-5 NAND vendor revenue nearly doubling in a quarter

The strangest confirmation is the hard-drive market. Nearline HDDs — the default home of cold data — hit 52+ week lead times, with Western Digital sold out for all of calendar 2026. The shortage actually widened the price gap flash was supposed to be closing: a 30TB QLC SSD cost 4.9x the equivalent HDD in mid-2025 and 22.6x by Q1 2026. By classical storage economics, QLC demand should have stalled. Instead it had its best quarter ever — hyperscalers building AI pipelines couldn’t get HDDs at any price, and paid a 22x per-terabyte premium for density, power (~30% less), and availability. When buyers pay that premium on allocation, the workload has changed underneath the market.

The QLC paradox: the SSD-vs-HDD price gap widened from 4.9x to 22.6x — and QLC demand boomed anyway

The Stack: Who Builds What

The signature of a real category shift is that every layer ships a product for it within the same 18 months. At the media layer, all five NAND makers are racing to 122–245TB QLC drives and purpose-built AI SSDs. At the systems layer, VAST Data raised at $30B in April 2026 on the back of a $1.17B CoreWeave commitment; WEKA’s flash-resident KV cache went GA; DDN — the storage behind xAI’s Colossus — took $300M from Blackstone at $5B; Pure Storage rebranded to Everpure, won a Meta design for its 150–300TB flash modules, and shipped a KV-cache accelerator in June; and NetApp logged ~500 AI wins in one quarter, more than its entire prior fiscal year.

The cloud tells the same story: S3 alone holds 500+ trillion objects, and per IDC, all-flash arrays crossed half of enterprise storage revenue for the first time in Q1 2026. None of this is a stock pitch; it’s a census. When a dozen vendors ship the same new product category inside two years — and NVIDIA writes the reference architecture legitimizing it — the workload is telling you what it needs.

The AI storage stack: who builds what, from NAND media through systems software to cloud object storage and NVIDIA’s platform layer

What Breaks the Thesis

Three real risks, in descending order. First, software keeps compressing the state: DeepSeek-style latent attention needs ~70KB per token against ~330KB for standard 70B-class attention, and KV quantization cuts another 2–4x. If compression outruns context growth, the spill to flash shrinks — though so far context and agent session lengths are growing faster than compression improves. Second, part of the NAND repricing is cyclical: BofA’s pre-melt-up forecast for 2026 NAND growth was +45%, not +281%, and the gap between those numbers is froth that can unwind — TrendForce already sees increases moderating to 10–15% in Q3. A 2027 normalization would deflate the headlines without touching the workload argument, but it would deflate them. Third, decode never touches flash — and if specialized prefill silicon (NVIDIA’s Rubin CPX pointed this direction) makes recomputing context trivially cheap, cached state loses value. Even that risk concedes the larger point, though: prefill economics now matter enough that NVIDIA designs separate silicon around them.

Summary

AI’s first infrastructure repricing was compute; the second was memory; the third is underway in storage, driven by a workload change rather than a shortage alone. Inference is stateful now — 42GB per long session, 5.7x context inflation per agent-hour, 45PB per hundred thousand users — and state that outlives a session must live on flash, because DRAM is too small, HDDs are unavailable, and the breakeven against recompute is roughly six reuses a month. The market has already internalized this in three places: the price sheets (90–99% cached-token discounts), the commodity (NAND up ~281% on record eSSD volumes and multi-year LTAs), and the stack (a KV-cache product from every serious storage vendor, with NVIDIA writing the reference architecture). For investors, the value sits at three tolls along the memory-storage boundary: whoever owns cache placement (NVIDIA and the systems software vendors), whoever supplies the flash it lands on (NAND makers, increasingly under long-term contract while capex stays disciplined), and whoever builds the silicon at the boundary itself (the new class of microsecond-latency AI SSDs). Instead of a watch list, here are three calls to hold me to. First: by end-2027, at least one major Western lab follows DeepSeek past a 95% cached-token discount — the arithmetic above says the economics already allow it. Second: by end-2027, every major cloud sells a named “context memory” tier the way it sells block storage today — NVIDIA’s G-tier taxonomy becomes product SKUs. Third: storage’s share of AI data-center spend roughly doubles by 2028, taken mostly from dollars that would have gone to HDD and cold tiers. If all three miss, the thesis was wrong.

Zoom out far enough and this is the oldest pattern in computing. Every wave — mainframes, databases, the web, cloud — eventually stopped being constrained by processing data and became constrained by moving it. AI looked like the exception because training really is compute-bound. Inference is the regression to the mean. This is the thesis this blog is named for: data has gravity. Compute is light and fungible; accumulated state is heavy and it stays put — compute exists to transform state, storage is state’s home, and silicon, networks, and price sheets all end up bending toward where it sits. Storage spent the first three years of the AI buildout as the boring layer. The state has arrived, and it needs somewhere to live.

This is the foundation essay of a series on data movement as AI’s real constraint. Coming next: the memory stack as the new compute stack (HBM → DDR → CXL → SSD → object storage), the rise of inference engineering, and why every AI company becomes a data infrastructure company.

Data Gravity covers AI infrastructure, compute economics, and durable software systems. Related coverage: Why KV Cache and Memory Drive AI Economics, The AI Memory Stack, and What Is CXL?.

1. Compute and memory have decoupled. Over 20 years, peak server compute grew roughly 3x every two years; memory bandwidth grew only 1.6x. Compounded, that’s a ~600x gap. It now decides how AI infrastructure gets built, not raw compute.

2. HBM is where the profit concentrates. It’s roughly half the manufacturing cost of an NVIDIA B200, and HBM plus packaging together are close to two-thirds of the chip. SK Hynix owns this tier today; Micron is closing fast; Samsung is recovering.

3. The scarcity cascades down. HBM’s premium pulled so much fab capacity out of ordinary DDR5 that commodity DRAM prices spiked 93–98% in a single quarter in early 2026.

4. Inference has its own bottleneck now: the KV cache — a model’s running memory of the conversation so far. A single 128K-context request already needs roughly a third of the memory the model’s own weights take up, and it multiplies with every concurrent user.

5. There is no single “memory stock.” There are structural winners at each tier — memory, packaging, connectivity, and the software that decides what lives where — and they are not the same companies.

Spending on HBM isn’t the same as owning the trade

“Buy more HBM” explains where the money is going right now. It doesn’t explain who gets to keep it.

AI systems are rebuilding the same memory hierarchy every computer has run on since the 1960s: small, fast tiers next to the chip, and progressively larger, cheaper, slower tiers further away. What’s different is scale. A GPU’s own on-chip memory is tens of megabytes. The cluster behind it needs petabytes. The data lake behind that needs exabytes. Every rung of that ladder is now its own market, with its own scarcity and its own winner.

That is the actual investment question — not “how much HBM capacity exists,” but which company owns which rung, and how durable is that ownership.

The memory wall stopped being theoretical

For 20 years, peak server compute grew roughly 3x every two years. Memory bandwidth grew only 1.6x over the same stretch. Compounded, that gap is why “the memory wall” went from an academic phrase to the constraint that has shaped cluster design since around 2023.

Moving data now costs more than computing on it — a DRAM read burns roughly 100x the energy of an on-chip cache read. That’s the physical reason every layer of the AI hardware stack is being redesigned around keeping data close, not just holding more of it. Compute stopped being the scarce resource. Feeding it is the scarce resource now.

The hierarchy, at a glance

Walk the ladder from a GPU’s own memory down to cold storage, and two things happen at once: capacity keeps growing, and speed keeps falling. That trade-off isn’t an engineering limitation to be solved — it’s physics. Nobody is building a memory technology that is simultaneously as fast as on-chip cache, as cheap as object storage, and as large as a data lake — the economics of stacking silicon mean nobody will.

The rest of this piece walks down that ladder and asks the only question that matters for capital: who owns each rung, and why is that ownership hard to take away.

Where the profit sits

HBM is the tier closest to the compute die, and it’s where the money actually concentrates. A B200 GPU carries up to 192GB of HBM3E delivering roughly 8 TB/s of bandwidth — enough that fitting it into the package now costs more than the compute die itself. Independent teardown estimates put HBM at roughly half of a B200’s total manufacturing cost, and HBM plus TSMC’s advanced packaging together near two-thirds of it.

That premium isn’t temporary, and it isn’t arbitrary. Stacking twelve memory dies on top of each other compounds defect risk with every layer, so yield is the real moat: SK Hynix’s proprietary stacking process reportedly reached 80%+ yield well before its competitors, and that lead is the entire reason it holds roughly 55–60% of the HBM market and crossed a $1 trillion valuation in 2026. Micron is closing the gap fast: fiscal Q3 2026 data-center revenue hit $41.5 billion, up 346% year over year. Samsung, after 18 months of failed qualification attempts, is finally shipping at scale.

None of it works without TSMC. Every HBM stack has to be packaged next to the compute die, and TSMC still can’t add that capacity fast enough: 2026 capex guidance was raised to $60–64 billion for it, and TrendForce still pegs the shortfall at about 10% even after that spending. Own HBM, or own the packaging step it depends on, and you own this tier. There isn’t a third way in.

The DRAM Crunch: Nobody priced this in

The scarcity doesn’t stop at HBM. Manufacturers spent two years pulling fab capacity toward HBM’s fatter margins, which starved ordinary DDR5 — the memory in every regular server and PC. Contract prices for plain DDR5 spiked 93–98% in a single quarter in early 2026, pushing total DRAM industry revenue past $97 billion in that same quarter.

CXL: A tier still proving itself

One rung down, CXL memory pooling is a real idea that hasn’t fully arrived yet. The pitch: let a server borrow memory sitting idle elsewhere in the rack instead of buying more. The waste it targets is real — roughly a quarter of provisioned DRAM sits unused at peak load. The catch is latency: pooled memory runs meaningfully slower than memory plugged directly into the server, so it’s a new, distinct tier, not a drop-in replacement for DRAM. Read our What is CXL piece?

It only started shipping in real hyperscale production in November 2025, when Microsoft Azure launched its first CXL-equipped cloud instances on Astera Labs’ controllers. Astera is the clearest public winner at this layer: FY2025 revenue hit $852.5 million, up 115% year over year. Its newest product line — fabric switching for AI racks — is on pace to become its largest by the end of 2026. The bet there isn’t on any particular memory technology. It’s on being the connective tissue between tiers, which is why it wins regardless of which memory generation or GPU architecture ends up on top.

KV Cache: Inference’s real bottleneck

Inference created a new kind of memory demand that barely existed three years ago. When a model answers a question, it keeps a running memory of the conversation so far — the KV cache — so it doesn’t have to recompute everything on every new word. At long context, it adds up fast: a single 128K-context request on a 70-billion-parameter model needs roughly 40GB of cache against the model’s own ~140GB of weights. Multiply that by the dozens of concurrent long-context sessions a production server actually holds, and cache — not the weights — becomes the largest consumer of GPU memory.

Architecture is a real moat here. DeepSeek-V3’s multi-head latent attention holds cache to about 70KB per token, versus 516KB for Llama-3.1-405B under standard attention — a 7.3x gap that’s pure architecture, unrelated to whose chips it runs on.

The rest of the industry is attacking the same problem a different way: moving the cache off the GPU and down the hierarchy, instead of paying to keep all of it in the most expensive memory available.

Moonshot AI's production system for its Kimi model, built this way, reports serving 75% more requests on the same hardware. This is the newest and least settled layer in the whole stack. Whoever builds the default software for deciding what stays on the GPU and what gets pushed down is building the operating system for the entire hierarchy — a bigger and more durable business than any single memory chip, because it sits on top of whichever hardware generation wins next.

Which data needs to live where

Every type of data an AI system touches ends up somewhere on this ladder, and the cost of misplacing it is asymmetric. Put something in HBM that didn’t need to be there, and you’re burning your most expensive tier on data that could have sat one rung down — a direct hit to margin on every inference call.

Who structurally wins

Ownership concentrates differently at each rung of the ladder.

The physical, scarce tiers — HBM and advanced packaging — are closed oligopolies with multi-year qualification cycles. That’s SK Hynix, Micron, and Samsung on memory, and TSMC on packaging. Micron has crossed $1 trillion in market value too, and TSMC sits near $1.86 trillion. Their sold-out capacity is a structural moat, not a cyclical one, and it’s the highest-conviction part of this thesis precisely because it’s already priced in.

The connective tissue between tiers is a separate, earlier bet. Astera Labs is the clearest public example, and the reason to like it is different: it doesn’t have to bet on which memory or GPU generation wins, because every generation still needs a physical link between them.

The placement software — the layer deciding, in real time, what data lives where — is the newest and least settled tier. It compounds independently of the hardware cycle underneath it, which is exactly what makes it worth watching over the next two years.

The winners criteria

The AI memory stack isn’t a single trade. It’s a ladder, and the money is spreading across every rung of it rather than pooling in one place.

• Own a physically scarce tier with real qualification barriers — HBM and advanced packaging reward incumbents, not new entrants.

• Own the connective tissue, not the memory itself — durable regardless of which hardware generation wins.

• Own the software that decides placement — the newest layer, and the one most likely to compound independently of the silicon cycle.

• Discount the assumption that memory intensity per token stays flat — the companies making memory less necessary are the underpriced counterweight to the companies making more of it.

Compute got its platform shift years ago. Memory just got its own — and it’s not one trade, it’s an entire supply chain with a different winner at every tier.

Inspired by a video by Robert from Anyscale — five reasons LLM inference broke the classic model-serving playbook, filmed on 10% battery after the mic died. Worth your 6 minutes and 40 seconds.

Three GPU clouds have now bought inference software instead of building it themselves. The latest deal landed this morning — so before anything else, congratulations to Robert and the entire Anyscale team: Nscale has entered a definitive agreement to acquire Anyscale, the company founded by the creators of Ray. Anyscale keeps its brand, its ~200-person team joins Nscale, and the deal is expected to close in H2 2026. My read: this becomes Nscale’s inference platform strategy. A company that owns power, data centers, and silicon just bought the layer that turns raw compute into tokens.

Sixteen months, three deals, one direction — up the stack:

The thesis behind all three: raw FLOPs are a commodity; delivered tokens per dollar are not. The margin between the two lives in software, because LLM inference is a genuinely different computational problem than the model serving that preceded it. Robert’s video is the cleanest six-minute explanation of why I have seen, and his five differences map neatly onto the life of a single request — how it gets admitted to the hardware (batching), how it executes (prefill and decode), what it remembers (the KV cache), where it lands (routing), and what shape the model underneath even has (sharding). This post works through each with the numbers attached. The list isn’t exhaustive — speculative decoding and quantization are their own posts — but these five are the reasons vLLM, SGLang, and TensorRT-LLM exist as a category, and the reasons that category keeps changing hands.

Difference #1: Variable-Length Computation & Continuous Batching

A ResNet doing image classification is a fixed-shape problem: fixed-size input, fixed FLOP count, fixed-size output. Batching is trivial — stack 64 images, run the graph, return 64 labels. LLMs break both ends of that contract. Prompts vary from 10 tokens to 200K, and outputs vary unpredictably because the model decides when to stop. Batch eight requests naively and the whole batch waits on the longest sequence while finished slots sit idle — you are burning HBM and FLOPs on padding.

The fix is continuous batching (iteration-level scheduling): the engine operates at the granularity of a single decode step, evicting finished sequences and admitting queued ones every iteration, so the batch stays full even as its membership churns. This is the single highest-leverage optimization in LLM serving. Anyscale’s own benchmarks showed up to 23× throughput over naive request-level batching — a vendor number and a best case, so discount it accordingly, but even the peer-reviewed vLLM results attribute 2–4× at equal latency to iteration-level scheduling and the memory management it enables. Either figure has no analogue in fixed-shape model serving. For a GPU cloud, this is the whole P&L in miniature: utilization is the denominator of cost per token, and every point of it recovered by the scheduler drops straight into the margin on delivered tokens.

Difference Two: Prefill Is Compute-Bound, Decode Is Bandwidth-Bound

LLM inference is really two workloads wearing one API. Prefill processes the entire prompt in parallel — thousands of tokens per matmul — and saturates the tensor cores. Decode generates one token at a time, and each step must stream the model weights and KV cache through HBM to produce a handful of FLOPs. The hardware ratio tells you everything: an H100 delivers roughly 989 dense BF16 TFLOPS against 3.35 TB/s of HBM3 bandwidth. Prefill sits comfortably above the roofline’s ridge; small-batch decode sits an order of magnitude below it. Same GPU, opposite bottlenecks.

Schedule both regimes on the same devices and they interfere: a long prefill stalls every decode stream sharing the GPU, which is exactly the inter-token latency jitter users feel. The structural answer is prefill/decode disaggregation — separate compute pools per stage, with the KV cache shipped between them — which lets you size, batch, and even select hardware independently for each regime (compute-dense parts for prefill, bandwidth-rich parts for decode). This is now table stakes in frontier serving stacks, and it is an orchestration problem, not a kernel problem — which is precisely why Ray sits underneath it, and why Anyscale was worth buying.

Difference 3: GPU Memory Management & the KV Cache

The KV cache is where LLM inference stops being a compute problem and becomes a memory problem. In practice, HBM capacity — not FLOPs — is the binding resource. Every attention layer stores key/value vectors for every token in context so they are not recomputed at each decode step, and the sizes are brutal even with grouped-query attention: a Llama-3-70B-class model carries about 320 KB of KV state per token — roughly 10 GB for a single 32K-token conversation (the full arithmetic is in the figure). A handful of long-context sessions can consume more HBM than the weights themselves, and the classic approach of pre-allocating contiguous max-length buffers wasted 60–80% of KV memory to fragmentation and over-reservation in pre-vLLM systems.

The breakthrough Robert references — vLLM’s PagedAttention — imports fifty years of operating-systems thinking: chop the cache into fixed-size blocks, map logical to physical blocks through a table, and share identical prefix blocks across sequences copy-on-write. The vLLM paper reported memory waste falling under 4%, which translated into 2–4× throughput at equal latency simply because more sequences fit in the batch.

Then comes the policy layer: which prefixes to retain in HBM, which to swap to host memory or NVMe, which to recompute. None of this exists in ResNet serving. All of it determines your cost per token.

Difference #4: Prefix-Aware Routing

Scaling classic model serving is a solved problem: replicate the model, put a load balancer in front, route round-robin or least-loaded. That works because every replica is interchangeable — no replica holds state that makes it a better destination for a given request. KV caching destroys that symmetry. Once replica 2 holds the cached prefix for your multi-turn conversation (or your agent’s 15K-token system prompt), sending turn six to replica 3 forces a full re-prefill of state that already exists in the cluster. The longer the shared prefix, the worse the miss: at agentic prompt lengths, prefill dominates time-to-first-token, so a cache hit removes most of the wait — routinely an order of magnitude on TTFT.

Prefix-aware routing makes the load balancer cache-conscious: hash the prompt prefix, track which replica holds which blocks, and trade off cache affinity against load. This matters more every quarter, because workloads are shifting toward exactly the shapes that reward it — multi-turn agents, shared system prompts, RAG templates — where the majority of input tokens are re-reads of tokens the cluster has already processed. It is also why the router is migrating from dumb infrastructure into the inference engine itself: the scheduler now needs a cluster-wide map of KV state to make a good decision. Routing became stateful, and stateful routing is software the neoclouds did not have.

Difference #5: Model Sharding for Mixture-of-Experts

The last break with the old playbook is the model’s own shape. Classic scale-out replicates the whole network on every GPU: N devices, N identical replicas. Frontier models made that impossible twice over. First they outgrew a single device. Then they went mixture-of-experts — attention layers interleaved with expert layers containing dozens to hundreds of feed-forward experts, of which only a few activate per token. DeepSeek-V3 is the canonical public example: 671B total parameters, ~37B active per token across 256 routed experts per MoE layer. The economics are the point — you buy frontier quality at a fraction of the per-token FLOPs — but only if the serving topology cooperates.

The standard deployment replicates attention across GPUs while sharding the experts between them (expert parallelism), then routes each token mid-forward-pass to whichever device holds its activated experts. As Robert puts it, you no longer have ten independent replicas — you have one big replica with a routing fabric inside it. Every decode step becomes an all-to-all communication event, expert load-balancing becomes a live scheduling problem, and the failure domain is the whole pod rather than one box. Since most frontier models are now MoE, this is the default serving problem, not the exotic one — and it is squarely an engine-plus-orchestration problem of the kind Ray was built for.

The Data Gravity Take

Pull the five threads together and the point is compounding. Continuous batching multiplies with paged memory, which multiplies with cache-aware routing, disaggregation, and expert-parallel sharding — none of these moves cost per token by 5%, and together they move it by integer multiples. That stack is the difference between a GPU cloud earning commodity rental yields and one earning platform margins on delivered tokens, and from there the M&A wave explains itself: CoreWeave took the developer workflow with Weights & Biases, Nebius took the optimization stack with Eigen AI for ~$643M, and today Nscale took Anyscale.

The obvious objection: vLLM and SGLang are open source, so what exactly is anyone paying $643M for? The answer is that the free part is the smaller part. An engine runs a model well on a node — that is where open source has already won. Everything above the node is not free at all: the cluster-wide KV map that makes routing decisions, the disaggregated prefill and decode pools, the autoscaling across heterogeneous fleets, the multi-tenant scheduling, and the small number of teams on earth who have operated all of it at production scale. That is the orchestration layer, and it is what these deals actually price. It is no accident that all three targets sit there rather than at the kernel level — and no accident that Nscale bought the company whose founders created Ray, the substrate the industry already uses to wire these five techniques together, rather than simply adopting an engine anyone can download.

For founders, the lesson is that the value line has moved up-stack: differentiation in inference now lives in schedulers, memory managers, and routers, not in access to silicon. For investors, watch the remaining independents in the orchestration layer — the strategic buyers have shown their hand three times in sixteen months, and every vertically integrating compute platform still missing this layer is a motivated acquirer. Gravity, as ever, pulls the software toward the compute.

Thanks again to Robert and the Anyscale team — for the video, for Ray, and for a decade of making distributed AI legible. Congratulations on the next chapter.

Six and a half years from filing to first token, and 57 cents of every dollar never reaches a semiconductor company. CoreWeave passed one gigawatt of active power this spring and runs it at $8.31 billion of annualised revenue, so the scale is real and it is operating today.

The cost is the easy half. The harder question is what happens to it afterwards, and that is where this got interesting.

The physical plant lasts fifteen years and is financed on paper of about that tenor. The silicon that generates the revenue to service that paper has an economic life of three to four years, and loses most of its pricing power inside each turn. Nearly every unusual structure in this market exists to manage that gap.

Pricing the output turned out to matter more than pricing the input. At the wholesale rate CoreWeave’s own disclosures imply, the campus returns 13.4% over fifteen years. Drop fleet utilisation from 90% to 70% and that becomes 1%. Not a crash, not a demand shock. An ordinary year with some idle capacity, against a silicon bill that arrives every four years regardless.

What follows is the whole stack, because the arithmetic is the argument. The worked example below is a quarter of a gigawatt, which is the size real projects actually get built at. Multiply by four wherever you like.

Three clocks

Start with why the physical layer takes so long. NVIDIA ships a new rack platform every twelve months. A building takes 24 to 36 months. Getting power takes four to seven years.

A hall specified for 130 kW racks in 2025 energizes in 2030, into a market shipping 600 kW racks. Most of what looks odd about this sector follows from that: operators overbuilding electrical headroom they cannot populate for years, the premium on already-energized industrial sites, the sudden interest in behind-the-meter gas, and the pricing power sitting with anyone who can shave a year off a schedule.

Hyperscaler capex will land between $600 and $725 billion this year. Dell’Oro has global data center capex crossing a trillion. Nobody thinks money is the constraint. Transformer lead times are past 160 weeks. Switchgear is booked through 2028. Of the 12 GW of U.S. capacity supposedly arriving in 2026, about a third is under construction. The rest is financed, permitted, and stuck.

Which is the useful way in. If capital is not the constraint, then the only way to understand this buildout is to price the things that are.

Aerial or elevated view: steel frame, substation pad, laydown yard in one frame. The laydown yard is the detail worth seeing. It stays for years.

What the $37.6 billion buys

Everything below refers to the same project. I derived the equipment counts rather than asserting them, so you can change an input and rerun it.

Two need a flag. The transformer bank has to be sized on the N-1 case, not the peak. At 313 MVA, losing one unit of a four-by-100 MVA bank leaves 300 MVA and the campus sheds load, so the units go to 120 MVA and N-1 carries 360. And the flow rate assumes 75% of heat leaves through liquid. The other 25% still needs air handling, so liquid-cooled halls still have CRAH units, and retrofit budgets that skip this come in low.

Schedule: 78 months from interconnection filing to first token. The four longest items on the critical path are all electrical. Note the sequencing that forces. The transformer order goes in around month 12, two years before the building it serves is designed in detail, often before a tenant signs. Speculative transformer orders are now normal, and there is a working secondary market in reserved factory slots.

Power

Eight voltage conversions sit between the transmission line and a GPU. Each is a separate industrial supply chain with its own oligopoly and its own way of failing.

The stage that surprises people is the last one. Everything to the left of the rack existed in some form in a 2015 data center. The 800 VDC shelf and the rack-level battery do not have an analogue, and they are there for a specific reason: a training collective makes tens of megawatts appear and disappear in milliseconds, and the distribution system has to absorb that without passing it upstream to the utility.

Land and the queue

Site selection used to weigh latency, tax abatement and fiber. Now it weighs one thing: how fast can this site get power. In practice developers screen first for an existing queue position, or an energized substation with a spare bay. Then transmission within a few miles, on two independent corridors. Then room for gas or batteries as a bridge. Then water rights. Then whether the local union hall can field a thousand electricians for two years.

PJM projects entering service in 2025 averaged over seven years: three-plus to get an interconnection agreement, then about four more waiting to energize. The bottleneck moved downstream. It is no longer the study, it is physical delivery of transmission and equipment.

The base rate is worse than people assume. Of all capacity that entered queues between 2000 and 2019, 13% had reached commercial operation by the end of 2024. Entering a queue is easy. Clearing one is not.

230 kV customer substation serving a data center, Quanta Services

Transformers and switchgear

A large power transformer is bespoke: grain-oriented electrical steel, copper windings, custom bushings, a test bay, and a heavy-haul route to site. You cannot flex that capacity on any useful timescale.

Eaton has guided to solid-state transformer orders in the second half of 2026, shipping late 2027. First real technology relief valve, and it arrives after this capex wave. Watch whether those orders show up as commercial-scale commitments from named hyperscalers or as pilots. The difference tells you whether it is real.

Switchgear is effectively sold out through 2028, with high-voltage breakers at 18 to 24 months. Eaton’s numbers show what that does to a P&L. Q1 2026 revenue was $7.45 billion, up 17%, on a $22.8 billion backlog. But data center orders were up 240% and data center revenue up 50%. When orders grow fourteen times faster than the corporate top line, mix is the whole story. Forgent Power Solutions has a nearly $2 billion revenue backlog, primarily across switchgear and other power infrastructure products, up 157% year over year.

Which raises a question the sector rarely answers directly: why do these lead times turn into margin rather than share loss? Because the layers are concentrated enough that nobody has to compete on price.

The top five suppliers control 62% of North American data center power. In transformers the concentration is higher still, capped by grain-oriented steel and test-bay capacity rather than by capital. Rack integration is the instructive counter-example: Foxconn and Quanta hold 65 to 70% between them and still run single-digit to low-teens gross margins. Concentration without pricing power is just concentration.

UPS, generators, busway

AI training does not draw power smoothly. It draws a synchronised square wave, tens of megawatts stepping on and off in milliseconds as an all-reduce completes across the cluster. That is a power-quality problem, not a capacity problem. Vera Rubin includes 20 times more on-rack energy storage than Blackwell purely to smooth it.

The bigger consequence gets missed. A training cluster restarts from checkpoint, so losing power costs you minutes, not data. Several AI-native operators have concluded that full 2N generator backup on training halls protects against a failure the workload already tolerates, and are building to lower tiers. On this campus that is the difference between about ninety 3.25 MW gensets and almost none. Call it $250 to $350 million and a year of procurement.

Busway is the one nobody was watching. Sightline Climate added it to the long-lead list for the first time in May 2026, which tells you the constraint now covers the entire path from substation to server rather than one point on it. Moving to 800 VDC pushes 150% more power through the same copper and removes some 200 kg of busbar per rack, about 200 tonnes across this campus.

Heat

Every watt into a GPU comes back out as heat. This campus is a 250 MW heater. Water carries about 3,500 times more heat per unit volume than air, so past 50 or 60 kW per rack, air stops being an option.

The water-versus-power trade decides more than it looks like it should. Evaporative cooling on this campus runs on the order of a billion gallons a year. Going fully dry eliminates that and costs about 30 MW of extra electrical load. At a fixed 300 MW service, 30 MW is about 140 Vera Rubin racks you no longer get to install. Water and compute are directly interchangeable, and in most 2026 markets water is the cheaper of the two.

The 45°C specification follows from the same logic. Chillers eat a fifth of facility power. A plant rejecting heat at 45°C instead of 18°C runs free cooling most of the year in most climates. Moving PUE from 1.30 to 1.12 on a 300 MW service frees about 40 MW, or 190 more racks, with no new interconnection and no new transformer. Best return on capital in the building.

Note who has been buying: Eaton acquired Boyd Thermal in March 2026, Schneider owns Motivair. The electrical incumbents bought into thermal because customers want power and cooling as one reference architecture. Vertiv already had that, which is much of why it holds the position it does.

Source: Mitsubishi Electric

Compute

Vera Rubin replaces the cable backplane with a PCB midplane, runs a liquid-cooled busbar at 50 V into the compute trays with DC-DC shelves stepping down from 800 V, and carries 20 times the on-rack energy storage. Kyber in 2027 turns eighteen blades vertical to fit 576 GPUs into one scale-up domain.

Slab-to-slab height, floor loading, pipe diameter and electrical room area are fixed when you pour. When you underwrite a colo operator in 2026, the design density of the newest hall tells you more about terminal value than current occupancy does.

Source: Nvidia

Networking, optics, storage

Three separate networks, and conflating them is the most common analytical error here. Scale-up is NVLink over copper inside the rack, 72 packages at 3.6 TB/s per GPU, under two metres. Scale-out is Ethernet or InfiniBand across the hall at 800G moving to 1.6T, almost entirely optical. Scale-across is coherent DCI between buildings, which exists because single-site power has capped out and a frontier run now has to span campuses.

The constraint underneath all three: an idle GPU is the most expensive object in the building. A $4 million rack stalling on a gradient all-reduce burns capital faster than any network saving recovers, so AI fabrics run near 1:1 subscription and eat far more optics per unit of compute than cloud ever did. This campus takes about 250,000 transceivers and 5,000 km of fibre. Call it $200 million of optics.

One structural oddity. In every previous optical cycle a speed generation ramped, plateaued and got replaced. This time 800G, 1.6T and 3.2T are ramping simultaneously. That holds demand across more of the supply chain and delays the inventory digestion that ended every prior cycle. Whether it survives 2027 and 2028 is the open question for the sector, and I have not seen a convincing answer either way.

Storage gets ignored and causes disproportionate pain, and the driver is checkpointing rather than capacity. A two-trillion-parameter model in bf16 carries about 4 TB of weights, 8 TB of fp32 master weights and 16 TB of Adam optimiser moments. Roughly 28 TB per checkpoint, and the cluster sits idle while it writes. Keeping that stall under ten seconds needs 2.8 TB/s of write bandwidth. That number sets the architecture. It also explains why parallel filesystem vendors keep showing up as gating suppliers on cluster bring-up.

Commissioning

Integrated systems testing runs eight to fourteen weeks and reliably finds control-sequence failures that no individual system test catches. A chiller that fails to restage after a generator transfer. A CDU that alarms during a voltage sag. A building management system fighting the electrical monitoring system. Then GPU burn-in, days of synthetic all-reduce hunting for marginal transceivers and badly seated cold plates.Assume three to nine months between mechanically complete and revenue generating, and discount capacity announcements accordingly. A delayed 60 MW facility costs about $14 million a month in foregone revenue. Operators will pay almost anything to compress that window, and commissioning agents have quietly become a constrained resource of their own.

What the output costs

$9.4 billion and six and a half years spent. An hour of GPU time costs this much to produce.

72,700 packages times 8,760 hours at 90% availability gives 573 million billable GPU-hours a year. Against that: $4.0 billion of IT on a four-year life, $5.4 billion of infrastructure on fifteen, 2,144 GWh at six cents, and O&M at 2% of infrastructure capital. Cash cost is $2.78 per GPU-hour.

Note the split, because it is the argument. Fifty-seven percent of the capital contributes 63 cents an hour. Forty-three percent contributes $1.74. The infrastructure is cheap per unit of output precisely because it lasts, and the silicon is expensive precisely because it does not.

What the output sells for

This is where most models of this business go wrong, including my first pass at it. There is no single price per GPU-hour. There is a decay curve, and it is steep.

GB200 rack-scale capacity currently rents between $10.50 and $27.04 per GPU-hour across tracked providers. H100, two and a half years old, sits at $2.43 to $2.63 on the neocloud tier. B200 and H200 fall in between, in vintage order. That is not noise around a mean. It is a curve.

The implication matters more than the numbers. Revenue per GPU decays at close to the same rate as resale value. The residual on the silicon is not a separate exposure that shows up when you sell the fleet. It is the same exposure, observed twice, and it starts eroding cash flow in year two.

One caveat before the returns math, because everything downstream depends on it. Those are published on-demand rates. A 250 MW campus does not sell on-demand at retail. It leases wholesale to a lab or a hyperscaler, and that realised price is not published. Rather than pick a number and pretend, I have expressed everything below as a percentage of the retail index, and then checked the answer against a company that discloses enough to derive it.

Does it earn the capital back?

Cash cost is not where anyone should stop. The question is whether the campus returns its capital across a full infrastructure life, with silicon replaced every four years and each new generation entering at the top of the decay curve.

At a 55% wholesale realisation, which implies a lifetime average around $4.04 per GPU-hour, the campus returns 13.4%. Breakeven against an 8% unlevered hurdle is 49% of the retail index, or about $3.62 average. Above that it works. Below it, it does not, and at 35% the project destroys capital outright.

But price is not the variable that decides this. Utilisation is. Moving from 90% to 70% fleet utilisation takes the IRR from 13.4% to 1.0%. Price would have to fall by a fifth to do equivalent damage. The reason is the treadmill: $4 billion of silicon has to be bought every four years whether the fleet is busy or not, and a half-idle fleet still pays for it in full.

Almost nobody discloses utilisation. If you take one operational metric away from this piece, make it that one.

At a 55% wholesale realisation, which implies a lifetime average around $4.04 per GPU-hour, the campus returns 13.4%. Breakeven against an 8% unlevered hurdle is 49% of the retail index, or about $3.62 average. Above that it works. Below it, it does not, and at 35% the project destroys capital outright.

But price is not the variable that decides this. Utilisation is. Moving from 90% to 70% fleet utilisation takes the IRR from 13.4% to 1.0%. Price would have to fall by a fifth to do equivalent damage. The reason is the treadmill: $4 billion of silicon has to be bought every four years whether the fleet is busy or not, and a half-idle fleet still pays for it in full.

Almost nobody discloses utilisation. If you take one operational metric away from this piece, make it that one.

Checking the model against something real

A 55% realisation is the load-bearing assumption in everything above, and I do not like load-bearing assumptions I cannot test. CoreWeave, as it happens, publishes both halves of the ratio.

CoreWeave reported $2.08 billion of revenue in Q1 2026, up 112%, and crossed one gigawatt of active power during the quarter. Annualised against that exit figure, revenue runs at $8.31 million per megawatt of active power per year.

Treat that as a floor rather than a midpoint. They crossed a gigawatt during the quarter, so the revenue was earned on a lower average base and the true figure per active megawatt is higher. Full-year guidance of $12 to $13 billion, against a ramp from 1.0 to more than 1.7 gigawatts, implies about $9.3 million.

My model produces $8.05 million per megawatt at a 55% realisation and $9.51 million at 65%. So the disclosure brackets a realisation of 55 to 65% of the retail index, which is an IRR range of 13 to 22% rather than the single 13.4% figure. I have kept 55% as the working assumption throughout because it is the conservative end, and because the honest position is a range.

Two things that check does not do, and I would rather say them than have someone else. CoreWeave leases a large share of its capacity rather than owning it, so its income statement carries rent where my model carries depreciation. Its 56% adjusted EBITDA margin is therefore not comparable to anything here. And its fleet spans Ampere through Blackwell, so the figure is blended across vintages. That makes it the right comparison for a lifetime average and the wrong one for year one.

What it does do is move the weakest input from assumption to a bounded range anchored on a filing. If you have better wholesale data, you can see exactly which number to replace and what it does to the answer.

One more number from the same filing. CoreWeave guides to $31 to $35 billion of capex this year against $12 to $13 billion of revenue, having raised the capex range mid-year. Capex is running at more than two and a half times revenue. That is what the treadmill looks like on a real cash flow statement.

So which depreciation schedule is right?

I said earlier that hyperscalers book five to six years and AI-natives book three, and that both cannot be right. That was too glib. Having worked the decay curve, I think both are right, and the reason is more useful than the dispute.

H100 SXM5 cards sold for around $40,000 in late 2023. They move for $6,000 to $15,000 now. Roughly 70% in about thirty months, and the decline is back-loaded: value holds through the first two years, then falls away as the next generation arrives in volume. A five-year straight line implies about 20% annual decline. The observed curve is nothing like that.

The mechanism is not mysterious. Inference on an H100 costs eleven times what it costs on a B300. Once the newer part ships in volume, the older one cannot be priced competitively for the workload that would otherwise keep it busy. The hardware still works. The economics stop.

This is the part I got wrong the first time. That argument is decisive for an operator whose revenue comes from the rental market. It is much weaker for one with internal demand.

A hyperscaler at year four cascades the fleet to search ranking, recommendations, ads inference and internal serving. None of that needs frontier performance, the hardware stays busy, and a five to six year life is defensible because there is a workload backstop that does not depend on rental pricing at all. A neocloud has no such backstop. Its year-four fleet competes in the open market against the H100 floor of $2.40 to $2.60.

The schedules are not wrong. They are being borrowed. Operators without internal workloads are applying useful-life assumptions that were derived by companies who have them, and lenders are underwriting against the borrowed number. That is a narrower claim than I started with and I think it is the correct one.

How it is financed

Which brings us back to the top. If the revenue-generating asset has a three to four year life, what is the debt behind it?

The scale here is easy to miss. Total data center debt issuance nearly doubled to $182 billion. Private credit lending to AI-related companies went from near zero to over $200 billion in a few years, and Morgan Stanley projects another $800 billion over the next two. JPMorgan expects $30 to $40 billion of annual data center securitisation across 2026 and 2027, which would be 7 to 10% of combined CMBS and ABS issuance.

Meta’s Hyperion is the template. A special purpose vehicle called Beignet Investor, arranged by Morgan Stanley, raised $27 billion of A-plus rated debt anchored by PIMCO and BlackRock, plus equity from Blue Owl. Meta keeps operational control, leases the campus back, and converts what would have been capex into predictable opex. The structure is elegant and it is being copied.

Where the money goes

Coverage of AI capex collapses almost entirely to accelerators. Accelerators are 43% of the bill. The other 57% goes to a supply base with multi-year backlogs and, for the first time in decades, real pricing power.

Contractor backlogs are the highest-quality real-time data in the sector, because backlog is signed contract booked 12 to 24 months before revenue. Comfort Systems ended June with $14.06 billion against $8.12 billion a year earlier. Same-store went from $8.12 to $13.70 billion. Q2 revenue up 50%, EPS up 92%, operating cash flow over a billion in a single quarter. Three straight quarters of same-store backlog growth above a billion, with duration running into 2027 and 2028.

On the multiples: Vertiv at 44 times forward is the purest expression of the thesis and leaves no room for a guidance miss. Eaton at 30 times looks slower only because data center is a minority of a very large book, so the 240% order growth is the more useful number. Comfort Systems and EMCOR trade on contractor multiples against utility-like visibility, and that is the clearest dislocation in the group. It is also the most exposed if conversion slips.

The portfolio implication is the part I keep coming back to. The suppliers to the physical layer have this pointed the right way round. They sell into a fifteen-year asset, they book backlog that converts over 12 to 24 months, and they carry no residual risk on the silicon. The operators and the neoclouds carry all of it. Owning the picks and shovels is not just a way to get AI exposure without picking a model winner. It is a way to be long the buildout while short the specific thing that makes the buildout fragile.

How this breaks

The efficiency shock is the one I actually worry about, not because it is likely on any particular timeline but because it is the only scenario where demand for megawatts falls while demand for AI rises. Nobody in this supply chain is positioned for that. Everything else on that list is a timing risk.

A demand air-pocket in the next four quarters is the one I take least seriously, and the backlog data is why. You cannot quietly cancel $14 billion of signed mechanical contracts. The contractors would report it before the hyperscalers guided to it.

The scenario the table understates is slower and duller. No collapse, just utilisation drifting into the low 70s while depreciation gets marked toward the secondary market. On the model above that combination takes the IRR from 13.4% to about 1% without a single dramatic headline, and it lands on the operators carrying SPV debt rather than on the suppliers.

What to watch

Everything in this piece reduces to a small number of things you can actually watch.

What this actually is

This gets framed as a software story funded by capital markets. It is closer to a heavy industrial construction story with a chip company at one end and a utility at the other, running through a supply chain that spent twenty years optimising for a demand curve that no longer exists.

Fifty-seven cents of every dollar in that $9.4 billion never reaches a semiconductor company. It goes to transformers with three-year lead times, switchgear sold out through 2028, chillers and CDUs and two hundred tonnes of copper busbar, and about 2,500 people spending two years on a site in Ohio or Louisiana turning capital into megawatts.

A gigawatt costs $37.6 billion, takes six and a half years, and 57 cents of every dollar of it is transformers, switchgear, chillers, copper and labour. That physical plant is the durable half. It costs 63 cents an hour to run, it lasts fifteen years, and nobody can build it quickly. The problem is that it is financed on the strength of revenue from the other half, which turns over five times before the paper matures and loses most of its pricing power inside each turn.

At the realisation rate CoreWeave’s own disclosure implies, the campus returns 13.4% and everyone is fine. At 70% utilisation it returns 1%. The distance between those two outcomes is not a market crash. It is an ordinary year with some idle capacity.

The buildings will be fine. Watch who is holding the silicon.

The short version

Ethernet didn’t outrun InfiniBand, it absorbed it. RDMA in 2010, adaptive routing in 2023, in-network collectives in 2025, each clone arriving faster than the last, until Ultra Ethernet started shipping capabilities InfiniBand never had. The share flip is done, from over 80 percent InfiniBand in late 2023 to roughly two-thirds Ethernet ever since 2025. NVIDIA’s answer was to sell the winner, and its networking business now runs near $60 billion annualized. Now the war is climbing into the rack, where AMD’s UALink-over-Ethernet systems launched this week, and the durable value is pooling at the endpoint, in the NIC, the rack-scale fabric, and a collectives software layer that nobody owns yet.

What the ending of a standards war teaches

Every large AI cluster is really two computers. The first is the one everyone talks about, the racks of GPUs. The second one decides whether the first is actually earning its keep. That’s the network stitching tens of thousands of GPUs into a single machine. When the second computer fails, the first one stops earning. At frontier scale something in the cluster breaks every few hours, and because training is synchronous, a single flapping optic or misbehaving NIC doesn’t degrade one server, it stalls all of them. On a 100,000-GPU cluster, roughly $4 billion of silicon renting for about $300,000 an hour at market GPU rates, a failure that takes thirty minutes to detect and checkpoint-restart burns $150,000 of compute. And a failure that never quite resolves, a marginal link retraining every few minutes, becomes a tax on every step of every epoch of every run.

That failure math is why the fabric stopped being a plumbing decision and became a boardroom one, and why the fight over it, between InfiniBand, the purpose-built interconnect NVIDIA acquired with Mellanox, and Ethernet, the fifty-year-old open standard, turned into the biggest open-versus-proprietary fight in infrastructure since Linux took the server. Here’s the thing though. That war has effectively been decided. NVIDIA’s own networking segment, the house that InfiniBand built, now runs at nearly $60 billion annualized and its growth is led by Ethernet products. So this piece is not really about who won. It’s about the mechanism by which the winner won, because that mechanism is about to run again one tier up the stack, and this time you can watch it with your eyes open.

One chart carries the whole argument. For twenty-five years, every technical advantage InfiniBand shipped eventually showed up on Ethernet. Track how long each one stayed exclusive and you get the war’s entire logic in five rows.

Three lessons hang off that chart, and they organize everything below. First, in networking, a proprietary advantage is a rental, not a moat, and the rent is set by how hard the feature is to standardize. Second, the clones arrive faster each cycle because the buyers switched sides, and the Ultra Ethernet Consortium is nothing more than the customers collectively funding the cloning of their supplier’s moat. Third, the moment the copier starts originating, the war is over and the front moves. Ultra Ethernet’s spray-native transport is that moment. The new front, the rack fabric and the endpoint, opened this week.

Three moats, three half-lives

InfiniBand held three real technical moats. Each one fell by a different mechanism, and each mechanism teaches you something about the next war, so it’s worth being precise about what actually happened rather than reciting the acronyms.

The first moat was latency, and it was never made of wire. A kernel-mediated TCP round trip costs tens of microseconds in context switches and buffer copies. InfiniBand’s RDMA did the same work in about one, a 40x gap that mattered enormously when a collective operation finishes only when the last message lands. But the moat was software, kernel bypass and a hardware transport, and software travels. When RoCE ported InfiniBand’s own transport onto Ethernet in 2010, the 40x premium collapsed to roughly 2x, and at the megabyte message sizes that dominate gradient exchange, 2x on first-packet latency is noise against bandwidth. The lesson is that physics moats are rare and most latency moats are actually code, which means they are one port away from evaporating.

The second moat was congestion behavior, and its fall is the one that matters most for what comes next. AI traffic is a small number of enormous, identical-looking flows, and classic Ethernet load balancing hashes whole flows onto single paths, so two elephants collide on one spine link while the rest sit idle. Meta’s published RoCE work documents the failure mode at production scale, and NVIDIA’s marketing pegs commodity ECMP at roughly 60 percent delivered bandwidth, a vendor number, but one whose direction nobody who runs these fabrics disputes. InfiniBand avoided this with central scheduling and credits. Ethernet’s first answer, PFC plus DCQCN, worked but ran on operational pain, pause storms and victim flows and heroic tuning. The answer that stuck is packet spraying, splitting every transfer across all paths and reordering at the destination, which is what Spectrum-X, Ultra Ethernet, AWS’s SRD and now MRC all do. Notice where the reordering happens. Not in the switch. In the NIC. The durable fix for Ethernet’s biggest weakness was to move intelligence out of the network core and into the endpoint, and that relocation is the hinge of this entire piece.

The third moat was computing inside the network. SHARP lets InfiniBand switches sum gradients as they flow through, roughly doubling effective allreduce bandwidth, and it survived nine years without an open equivalent because it requires co-design across switch silicon, NIC and the collectives library. That is the longest half-life on the chart, and it explains NVIDIA’s entire subsequent strategy. When your co-designed features are the only ones that hold, you push co-design deeper, which is exactly what the NVL72 rack is. The lesson reads both ways. Co-design buys the longest exclusivity, and even it expires, since UEC 1.0 specified In-Network Collectives and Broadcom’s Tomahawk Ultra shipped collective offload in 2025.

The endgame came fast once the buyers organized. The Ultra Ethernet Consortium, founded in 2023 under the Linux Foundation by AMD, Arista, Broadcom, Cisco, HPE, Intel, Meta and Microsoft among others, shipped its 1.0 spec in June 2025, and it doesn’t imitate InfiniBand so much as leapfrog it. UET sprays packets across every viable path, delivers out of order while completing in order, starts transmitting before handshakes finish, and trims congested packets into a fast-lane loss signal instead of dropping them silently, targeting fabrics of a million endpoints. Then in May 2026 the model labs took the pen themselves. MRC, from OpenAI, Microsoft, Broadcom, AMD and NVIDIA, grafts the same ideas onto existing RoCE across eight parallel data planes, and its geometry is the real tell. It supports 131,072 endpoints in two switching tiers instead of three, three hops instead of up to seven, and double the compute for roughly 20 percent more switches. Meanwhile the proof of practice had already landed, since Meta built twin 24,576-GPU clusters as an explicit A/B test, Quantum-2 InfiniBand against RoCE on Arista, and trained Llama 3 on the Ethernet one.

The receipts are in the market data. In late 2023 InfiniBand carried over 80 percent of AI back-end switch spend. Ethernet crossed half during 2024, took more than two-thirds for full-year 2025, and held that share through 1Q 2026 even as InfiniBand sales more than tripled year on year on the Blackwell Ultra ramp. Both fabrics are growing absolutely. Only one is growing structurally, and the vendor behavior underneath tells you which, since Celestica retook the top Ethernet spot, Cisco posted the quarter’s biggest share gains, and 1.6T switches are sampling for an H2 ramp, the kind of multi-vendor churn InfiniBand’s single-supplier structure cannot produce.

NVIDIA’s counter was to sell the winner

Watch what the incumbent did when it saw the chart above forming, because it’s the cleanest execution of the losing-a-standards-war playbook on record. In 2023, with Meta already on RoCE and AWS never having adopted InfiniBand at all, NVIDIA launched Spectrum-X, its own AI Ethernet, pairing the Spectrum-4 switch with a mandatory BlueField SuperNIC and doing per-packet adaptive routing with reordering at the endpoint. InfiniBand techniques in an Ethernet frame. It claims about 95 percent effective bandwidth against roughly 60 for commodity ECMP Ethernet, vendor numbers worth your skepticism, but the deployments are not marketing. xAI’s Colossus stood up 100,000 H100s in 122 days on Spectrum-X and has since grown past a reported half-million GPUs, still on Ethernet. Within two years Spectrum-X passed a $10 billion annualized run-rate and IDC ranked NVIDIA the number one datacenter Ethernet switch vendor by revenue, ahead of Cisco and Arista, in a market running above a $60 billion annualized pace.

The strategic content of Spectrum-X is not the throughput claim. It’s the attach. You can leave InfiniBand and NVIDIA still sells you the switch, the software, and above all the SuperNIC, because the performance features only light up end to end on NVIDIA endpoints. The skeptic’s line is that Spectrum-X is open the way a hotel minibar is convenient, and the skeptics are right, which is the point. NVIDIA read where the intelligence was moving and made sure that even its concession to open standards kept the endpoint proprietary. When the same company then co-authored MRC and opened NVLink Fusion to partners like Marvell, the pattern became unmistakable. Every NVIDIA response to openness protects the same square on the board, the endpoint.

The war doesn’t end, it moves up a tier

Here’s the map that explains why nobody gets to relax. An AI datacenter runs three networks, and they differ by roughly 40x in bandwidth per GPU at each boundary. Inside the rack, the scale-up domain moves 1.8 TB/s per GPU over copper NVLink and the rack behaves like one accelerator. One tier out, the scale-out back-end gives each GPU a 400 or 800G port into the Clos fabric, and that middle band is where the entire war just described was fought. The front-end tier below was always Ethernet and never contested.

Whoever owns a tier taxes everything that crosses it. Every fabric vendor is now trying to move up one band.

The second front opened on schedule, and this week it went live. AMD launched Helios on July 20 with Microsoft committing to deploy it at scale on Azure, a 72-GPU MI455X rack whose scale-up fabric tunnels UALink over Ethernet on switch silicon co-designed by Broadcom and HPE’s Juniper, with Pensando Vulcano UEC NICs handling scale-out and mass production ramping toward Q2 2027. Run the absorption chart’s logic against it. NVLink’s scale-up exclusivity dates to 2014-era SLI lineage but its modern form to 2016, the open clone arrived as a spec in April 2025 and as shipping hardware in July 2026, and NVIDIA pre-emptively opened NVLink Fusion to partners rather than wait to be cloned. The half-lives aren’t just shrinking. The incumbent now starts the absorption itself, which tells you it believes the pattern too.

The tier above the rack opened at the same time. Frontier training now spans buildings and metros, so NVIDIA shipped Spectrum-XGS for what it calls scale-across, with CoreWeave first in line, and Broadcom built Jericho4 for the same job. The corroboration that matters comes from Arista, which expects scale-across to contribute at least a third of its $3.5 billion 2026 AI target after describing the segment as virtually nonexistent a year ago. Inference pushes in the same direction, since disaggregated prefill and decode keep the heaviest traffic inside the rack and turn the middle tier into KV-cache plumbing that plain Ethernet handles fine. Both ends of the stack are pulling value away from the tier InfiniBand just spent three years defending.

Follow the money to the endpoint

Put every public company selling into this war on the same annualized basis and the growth ranking is itself an argument. NVIDIA’s networking segment runs near $60 billion annualized, up 199 percent, growing faster than its compute business. Broadcom’s AI semiconductors run at roughly $43 billion annualized, up 143 percent, with guidance implying a $64 billion pace this quarter and a backlog above $30 billion. Credo, which sells reliability-first copper cables into hyperscale racks, tripled fiscal 2026 revenue to $1.3 billion. Arista grew 35 percent and guided 2026 to $11.5 billion. Cisco’s datacenter switching, the pre-AI networking economy, grew 9. Sum the disclosed pieces, NVIDIA’s networking line, the roughly 40 percent of Broadcom’s AI revenue that is networking, Arista’s AI business and Credo, and AI networking grosses roughly $80 billion annualized before counting optics, cables, private vendors or the white-box builders. Two honest deductions apply, since the fiscal calendars don’t align perfectly and some of Broadcom’s switch silicon ships inside Arista and white-box systems and gets counted twice, so call the clean number $70 billion and change, still well over double what the entire datacenter switch industry grossed in a year as recently as three years ago. A component category became an industry in thirty months.

Now look at composition, because two details in those reports carry the thesis. Broadcom says networking is about 40 percent of its AI revenue and guides that mix toward 30 as custom accelerators ramp, which means the switch is becoming the smaller half of the AI story at the merchant switch champion itself. And the arithmetic of MRC makes the same point at the fabric level. When an MRC-style network doubles its GPUs, NICs double with them, one per accelerator, while the flattened two-tier topology needs only about 20 percent more switches. The endpoint scales with compute. The core deliberately doesn’t. That’s not a side effect, it’s the design goal of every post-2023 fabric, and it redraws where the revenue pools.

Size the two prizes and the migration stops being abstract. Call accelerator shipments 6 to 8 million units a year by 2027, across NVIDIA, AMD and the custom XPU programs, with one AI NIC apiece at $1,500 to $2,500 as 800G becomes the floor. Those are my assumptions, not a vendor’s, and they put the AI NIC pool at $10 to $20 billion a year and compounding linearly with compute, against a back-end switch market that Dell’Oro’s $100 billion five-year forecast averages to roughly $20 billion a year growing sublinearly by design. The endpoint is on course to match the core in dollars before the decade ends, and that’s before scale-up endpoint silicon, which Helios just turned into a merchant category.

The same migration is visible in the physical layer. A scale-out fabric consumes up to six optical transceivers per GPU across its tiers, Ethernet optics grew about 70 percent in 2025 on top of a doubling in 2024, and LightCounting sees a path to $100 billion a year in AI-cluster optics by 2030. Reliability, not just bandwidth, is what’s being priced, since at one interruption every few hours a flapping transceiver is a six-figure event. That’s why Credo can triple revenue selling copper cables whose pitch is a claimed thousandfold reliability edge over pluggable optics, and why NVIDIA is moving optics into the switch package entirely with co-packaged Quantum-X and Spectrum-X Photonics parts arriving through 2026, claiming 3.5x better power per bit. Everyone in the physical layer is selling the same two things now, joules and uptime.

Which brings the argument to its destination, the M&A record, because the buyers have been marking this thesis to market for fifteen years.

Every layer of that table has consolidated except one. The collectives software sitting above the NIC, the layer NCCL occupies, has no independent owner, and understanding why it’s empty tells you why it won’t stay that way. Until roughly last year there was nothing to be neutral about. Fleets were homogeneous, the fabric vendor shipped the software as a control point, NCCL from NVIDIA, RCCL as AMD’s port, and the only buyers rich enough to fund an alternative built private ones instead, NCCLX at Meta, MSCCL at Microsoft. Great abstraction layers only emerge when the hardware underneath fragments. VMware needed commodity server sprawl, Terraform needed multicloud. AI fabrics crossed that line in the last twelve months, with UEC NICs shipping from three vendors, UALink racks arriving alongside NVL72s, and neoclouds running InfiniBand and Ethernet estates side by side. An operator buying Helios and Vera Rubin systems in the same year now owns a fabric zoo with no common software plane. That is a company waiting to exist. Sizing it honestly requires an analogy rather than a bottom-up, because the category doesn’t exist yet. Infrastructure software that manages a hardware layer has historically captured low single digits of the spend beneath it, Datadog runs at roughly one to two percent of the cloud bills it watches, VMware at its peak took closer to ten percent of the server market it abstracted. Two to four percent of software attach on a $70 billion fabric complex is a $1.5 to $3 billion annual revenue pool by the decade’s end, concentrated in whoever wins the neocloud and enterprise tier first, and the founder pool to watch is the short list of people who have actually shipped NCCL, NCCLX or MSCCL.

The layer comes with a data business attached, which is what makes it venture-scale rather than a feature. A frontier cluster takes an interruption every few hours, Meta logged 466 across 54 days of Llama 3 training, and attributing a stalled step to a flapping link, a mistuned rail or a failing NIC is still done with vendor tools that stop at each vendor’s border. The collectives library is the one component that observes every transfer on every fabric in the job. Whoever owns it owns stall attribution, and in a market where an hour of cluster time costs $300,000, stall attribution isn’t a dashboard, it’s a claims adjuster for the most expensive machines ever built.

Now stress-test that thesis the way the steelman below stress-tests InfiniBand, because it can fail three ways. Hyperscalers might in-house the layer, the way they in-housed the NIC itself, since Annapurna became EFA, Google built Falcon, and NCCLX and MSCCL already exist, which would shrink the independent opportunity to neoclouds and enterprises. NVIDIA might integrate it away, folding enough fabric-awareness into NCCL and enough openness into NVLink Fusion that neutrality stops being worth paying for. Or the heterogeneity could prove temporary, since if Rubin-era racks simply win the next two years, fleets re-homogenize and the abstraction moment passes the way OpenStack’s did, and silicon cycles run longer than venture timelines either way. My honest read is that this layer gets built three times, once in-house, once by NVIDIA, and once independently, and the independent version matters only if open endpoints ship at volume on schedule. Which is exactly what the watchlist is for.

The steelman and the scoreboard

The bull case for InfiniBand from here rests on four legs, and it’s a real case. Quantum ships in lockstep with each GPU generation, which is why its sales tripled on the Blackwell Ultra ramp two years into Ethernet’s supposed rout. In-network collectives are specified in Ultra Ethernet but proven only on Quantum, and frontier training pays for that maturity gap today, not in 2027. The next million buyers are enterprises and sovereigns without elite network teams, a population that buys reference architectures that work out of the box, which is how CoreWeave built a business on Quantum fabrics. And co-packaged optics lands on Quantum first in NVIDIA’s own roadmap. The reference-architecture demand is measurable, not anecdotal, since of the 45 systems new to the November 2025 Top500, 31 chose InfiniBand and only 4 chose RoCE. Declining to a third of a market that quintuples is still a growth business. The share data says the war is decided, not that the loser disappears.

A thesis you can’t falsify is a mood. Six markers over the next eighteen months, each phrased so it can prove me wrong.

InfiniBand won the opening battle because it was the only fabric that treated the network as part of the computer. Ethernet won the war for the oldest reason in infrastructure. Given enough volume, open and good-enough runs down proprietary and perfect, and then stops being merely good enough. The next war, for the rack and the endpoint, started this week with the same armies in the same formation. This time, at least, we have the pattern.

Sources & further reading

Market data · Dell’Oro, 1Q 2026 AI scale-out report (Jun 2026) · Dell’Oro, “Ethernet is Winning the War Against InfiniBand” (Jul 2025) · Dell’Oro, 2025 full-year · Dell’Oro, $100B five-year forecast · IDC on NVIDIA taking #1 in DC Ethernet switching
Public financials · NVIDIA Q1 FY2027 results (May 2026) · Broadcom Q2 FY2026, $10.8B AI revenue (Jun 2026) · Arista Q1 2026 results (May 2026) · Credo FY2026 results, revenue triples (Jun 2026)
Standards and transports · UEC 1.0 announcement · VIAVI on UE 1.0 internals · The Next Platform on MRC (May 2026) · NVIDIA on RDMA and RoCE history · SHARP in-network computing
Deployments · Meta’s GenAI infrastructure · Meta SIGCOMM ‘24, RDMA over Ethernet at scale · Llama 3’s 466 training interruptions · Spectrum-X at xAI Colossus · Colossus at 2 GW and 555K GPUs · Microsoft Fairwater
Endpoints, optics and scale-up · CNBC on the AMD Helios launch (Jul 20, 2026) · Helios UALink-over-Ethernet details · Broadcom Thor Ultra · AMD Pollara · NVIDIA’s ~$900M Enfabrica deal · LightCounting on $100B AI optics · NVIDIA co-packaged optics · Spectrum-XGS · June 2026 Top500

• The frontier is no longer proprietary. Kimi K3 scores 57.1 on the Artificial Analysis Intelligence Index, #3 overall behind Claude Fable 5 (59.9) and GPT-5.6 Sol (58.9). The weights ship July 27. The last release cycle closed 13 points of gap in three months; 2.8 remain.

• Margin gravity: closed labs run API gross margins estimated above 80%; open-model serving clears at ~40–45%. On a ~$250B 2030 inference market, that 35-point spread is ~$90B a year of margin migrating from model IP to compute and serving.

• Bearish for labs: US labs’ OpenRouter token share fell ~70% → ~30% in twelve months. The moats that remain — product harnesses and internal checkpoints — are real, but nothing in the usage data prices them yet.

• The caveat: K3 is a token wastrel: $0.94 per benchmark task vs. $0.55 for GPT-5.6 Terra and $0.31 for Grok 4.5. The true Sputnik is a token-efficient open frontier model. Not yet.

• Bullish for compute, memory and serving: CoreWeave backlog $99.4B, Oracle RPO $638B, 2026 capex ~$725B — and export controls route open-model demand onto Western GPUs. Fireworks just repriced from $4B to $17.5B in seven months.

• Five falsifiable calls at the end, starting with: K3 serves below $10 per million output tokens within 30 days of the weights dropping.

What Actually Shipped

The frontier became a file

imi K3 is a 2.8-trillion-parameter mixture-of-experts model: 896 experts, 16 active per token, roughly 50B active parameters, a 1M-token context window, weights trained natively in 4-bit precision. On Artificial Analysis’ independent index it scores 57.1, third overall behind Claude Fable 5 (59.9) and GPT-5.6 Sol (58.9). At launch it took #1 on the Frontend Code Arena at 1,679 Elo, above Fable 5’s 1,631 — the first Chinese model to top a major coding leaderboard.

The weights go public on July 27. That sentence is the entire story, because it converts the frontier from a service into a file. A service has pricing power. A file has distribution.

The producer economics explain why this keeps happening. Moonshot’s K2 Thinking was reported by CNBC to have cost roughly $4.6M to train — the company calls the figure unofficial — against Stanford-estimated compute costs of $78M for GPT-4 and $191M for Gemini Ultra. The velocity is the tell: six frontier-class releases in twelve months, ARR doubled from ~$100M to ~$200M in roughly six weeks this spring, valuation up from $2.5B in early 2024 to $20B in May with a $30B round reportedly forming ahead of a Hong Kong IPO.

K3’s price is its own signal. At $3 per million input tokens and $15 per million output, it costs more than 3x its predecessor. The era of super-cheap Chinese AI is ending. The era of free frontier weights is not — Moonshot is telling you open intelligence now clears at closed-tier prices.

Margin Gravity

Sizing the pool that moves

The cleanest frame for what K3 means comes from Gavin Baker. A world with two or three dominant frontier labs earning 90%+ inference margins is a world where those labs become monopsonies: the price-setting buyers of power, data centers, and semiconductors. They vertically integrate into every layer beneath them and subsume the application layer above. Awesome for those labs. Net negative for everyone else in the economy.

“A world where there are only 2–3 dominant frontier labs with 90% inference margins is net negative for every other layer while being awesome for those 2–3 labs… Anything that lowers margins and increases competition at the model layer is good for every other AI layer: power, semiconductors, hyperscalers, neoclouds and yes even software.”

Gavin Baker (@GavinSBaker), on Kimi K3 as “an important inflection point for AI,” X, July 2026

Now put numbers on it. The 90% figure is not hypothetical: SemiAnalysis estimates Anthropic’s API gross margins above 80%. When the same workload runs on an open model through an inference platform, the model toll disappears and the serving layer keeps an estimated 40–45% (Tom Tunguz pegs Together at ~45%, Groq at ~40%). The compute underneath is identical. Baker’s point, exactly: an open model requires the same silicon as a closed model of the same size and architecture. Only the toll changes hands.

Scale that spread. Forecasters put AI inference around $250B a year by 2030. On that base, every point of model-layer gross margin is $2.5B a year. The spread between an 80%-margin closed API and a 45%-margin open serving stack is 35 points: roughly $90B a year of margin that lands in compute, serving, and silicon instead of model IP — if workloads migrate. That is the pool K3 puts in play. Not revenue. Margin.

There are two roads to this repricing, and both are being paved at once. Open weights at the frontier is one: that is Moonshot. The other is vertically integrated players who don’t care where the margin dollars land: Grok 4.5 pricing output at $6 per million tokens, Meta’s Muse Spark 1.1 at $4.25, Gemini running on Google’s own TPUs. This is also why Jensen Huang champions open source, and it is not sentiment. A diffuse model layer dissolves the monopsony before it forms — and K3 is the first time the open road reached the frontier’s doorstep.

Bearish: The Labs

The frontier premium now depreciates in months

The closed labs sell exactly one thing at those margins: the gap between their best model and the best model you can download. That gap used to depreciate over years. K2.6 to K3 closed 13 index points in three months; 2.8 remain. The premium is still real. Its half-life is now measured in release cycles.

The market has been voting for a year. US labs’ share of tokens routed through OpenRouter fell from roughly 70% in June 2025 to ~30% in June 2026. DeepSeek alone is now the largest model family on the platform at 16.3% of volume.

Set that against the leader’s cost structure. OpenAI booked $13.1B of 2025 revenue against a reported $38.5B net loss, and carries $1.4T in compute commitments. That is a business that must defend premium pricing for a decade, at the exact moment premium pricing became optional for a growing share of workloads.

The labs’ own capital allocation concedes the point. OpenAI is designing 10GW of custom accelerators with Broadcom and bought 6GW of AMD supply. Anthropic contracted up to a million Google TPUs and committed $30B to Azure. Vertical integration is what you do when you expect the model layer’s margin to stop paying for the buildout.

Baker calls K3 only potentially negative for OpenAI and Anthropic. He grants them two outs, and both deserve to be taken seriously.

First, the product may now matter more than the model. Claude and ChatGPT are harnesses: memory, tooling, distribution, enterprise trust. Anthropic’s reported $47B run-rate is a workflow story sitting on top of models, not token resale — and it holds 61% of gateway spend on 32% of gateway tokens, which is what a working moat looks like. Second, the labs may hold internal checkpoints well ahead of anything public, compounding through recursive self-improvement. A few months of lead there could become permanent.

Both are real. Neither shows up in the direction of the usage data.

The dollars-versus-tokens gap is how you watch the race. Menlo Ventures put enterprise workloads at 89% closed at the end of 2025. Vercel’s gateway shows open-weight models carrying 29% of tokens and under 4% of spend, with token share nearly tripling in two months. Tokens lead. Dollars follow, unless the harness moat holds.

The Token-Efficiency Caveat

K3 is a token wastrel, and that cuts both ways

Intelligence per dollar is the product of two terms: price per token, and intelligence per token. K3 wins the first and loses the second.

It burned 132M output tokens running Artificial Analysis’ evaluation suite, more than double the ~63M median, because it ships with always-on maximum reasoning and no effort controls yet. The result is $0.94 per benchmark task: cheaper than GPT-5.6 Sol ($1.04), half of Claude Opus 4.8 ($1.80), but 71% above GPT-5.6 Terra ($0.55) and 3x Grok 4.5 ($0.31). The rough edges don’t stop there — K3’s hallucination rate on AA’s Omniscience test worsened to 51%, from K2.6’s 39%.

So the discount is real at the top of the closed stack and absent against its efficient tier. Running AA’s full index cost ~$2,700 on K3 versus ~$6,200 on Claude Fable 5, the most expensive model AA has ever benchmarked — but a closed lab routing to a token-efficient model at Terra’s price still undercuts the open alternative on delivered work. This is why Baker calls K3 an inflection point rather than a Sputnik moment. The true Sputnik is an open frontier model that is also token-efficient. It has not happened yet.

But inefficiency cuts both ways. Every wasted token is billed compute. A token-inefficient open model is a subsidy from Moonshot’s margin to whoever owns the GPUs it runs on.

Bullish: The Compute Layer

Free models are demand insurance for GPUs

A free frontier model destroys margin in exactly one place, the model API, and creates load everywhere beneath it. K3’s weights are ~1.4TB even at 4-bit precision. Moonshot recommends supernodes of 64 or more accelerators to serve them. Nobody downloads that onto a laptop.

This is Jevons operating at infrastructure scale. As the price of intelligence collapses, consumption explodes, and every unit of it clears through someone’s data center — including K3’s doubled token burn. Google now processes 3.2 quadrillion tokens a month, up 7x in a year. Inference has gone from a third of AI compute in 2023 to a projected two-thirds in 2026.

The operators are printing the evidence. CoreWeave grew Q1 revenue 112% to $2.1B with a $99.4B backlog. Nebius grew 684% and signed $17.4B from Microsoft and $27B from Meta. Oracle’s RPO hit $638B, up 363% year over year. Hyperscaler capex guidance for 2026 stands near $725B, up ~77%, against McKinsey’s $6.7–7T buildout estimate through 2030.

Memory is the purest expression of the trade. K3 activates less than 2% of its 2.8 trillion parameters per token, but every one of its 896 experts must sit resident in memory: 1.4TB before it serves a single request. Sparse models economize on FLOPs and gorge on capacity. Long context makes it worse — a 1M-token window is KV cache, and KV cache is memory. Verbosity makes it worse still: K3’s doubled token burn is decode-heavy work, and decode is bound by memory bandwidth, not compute. Open, sparse, long-context, token-hungry models are precisely the workload HBM was built for. The market is already pricing the squeeze: Bank of America puts the HBM market at $54.6B in 2026, up 58%, with DRAM revenue up 51% and total memory past $440B — and Microsoft attributed roughly $25B of its capex guidance to memory and component inflation alone. Margin gravity says the dollars pool at the scarcest physical constraint. In 2026 that constraint is not FLOPs. It is memory.

The geography is the under-priced part. Export controls mean Moonshot cannot serve global demand from Chinese silicon; it serves K3 today at a throttled ~26–28 tokens per second. Releasing the weights exports the serving demand to the only infrastructure that can absorb it: Western GPUs. The Sputnik analogy gets the consequences backwards. Sputnik was a rival’s closed capability. K3 is a demand shock delivered directly to American data centers.

Bullish: The Inference Companies

When the model is free, serving is the business

Inference providers are the direct monetization layer for open weights. They capture the serving revenue Moonshot gives up, and the private market has spent twelve months repricing them for it.

Fireworks closed a $1.5B Series D at $17.5B this week, up from $4B in late 2025, with ARR past $1B (5x YoY), daily token volume past 40T, and ~95% of tokens served on models fine-tuned from open weights. Together raised at $8.3B on more than $1.15B of bookings, serving open models exclusively. Baseten’s estimated ARR tripled to ~$600M in a single quarter. Nvidia agreed to pay ~$20B for Groq. Cerebras doubled on its IPO day.

Now watch the mechanism run in real time. K3 is currently served by exactly one provider: Moonshot itself. July 27 changes that. When K2.6’s weights opened, it hit #1 on OpenRouter with 1.88T tokens served in a week across roughly twenty competing hosts; when DeepSeek’s last flagship dropped, Fireworks listed it the same day. Hosts profitably served K2.7-class models (32B active parameters) at $4 per million output tokens. K3 activates 50B parameters and lists at $15. That spread is the auction’s opening bid, and competition will close it toward cost — margin that accrues to serving platforms and their GPU suppliers, not to Moonshot, which collects nothing and has already seeded vLLM with the attention kernels to accelerate its own disintermediation.

Who Wins & Why

The margin ledger, layer by layer

Call it margin gravity. In AI, margin flows down the stack until it pools at the scarcest physical constraint. Open frontier weights unbundle intelligence from the infrastructure that serves it — and what cannot be downloaded is gigawatts of contracted power, installed accelerators, serving economics won through utilization and kernels, and the enterprise control plane.

What to watch — five dated catalysts

1. July 27: the weights and the license. Confirmation of modified-MIT terms, and how fast third-party hosts light up. Precedent says same-day: Fireworks listed DeepSeek’s last flagship within hours of the drop.

2. A token-efficient open frontier model. Baker’s true Sputnik moment. K3 at $0.94/task vs. Grok 4.5 at $0.31 defines the gap; the first open model to close it ends the closed labs’ last economic argument below the frontier.

3. Grok 5, Composer 4, Muse 2 on the Pareto frontier. Vertical integration is the second road to model-layer margin compression. Multiple integrated players at multiple price-intelligence points would finish what open weights started.

4. The tokens-to-dollars gap. Open weights: 29% of gateway tokens, under 4% of spend. Each quarterly print that narrows it converts the token vote into revenue for inference providers and neoclouds — and out of lab pricing power.

5. How fast the labs become their own infrastructure. OpenAI’s 10GW Broadcom program and 6GW of AMD; Anthropic’s million TPUs and $30B Azure commitment. The faster the labs verticalize, the louder they are conceding that the model layer alone no longer carries the valuation.

The risk case is concentrated, not diffuse. Enterprise dollars stay closed longer than the token data implies — trust, liability, inertia. Washington moves: the House already probed Airbnb and Cursor’s parent over Chinese-model use, and restrictions in regulated sectors would slow adoption where the dollars are. Or a lab’s internal checkpoints compound into a lead nobody closes. Each slows the repricing. None reverses it, because the mechanism — margin flowing to the scarcest physical constraint — doesn’t depend on which model wins.

The weights are free. The gravity is not.

• One Blackwell GPU costs Nvidia roughly $6,400 to build — 45% of that is HBM memory alone — and sells for $30,000–$40,000. The 5–6x gap between those two numbers, captured entirely at the design layer, is the single clearest illustration of where value in this stack actually accrues.

• Six buyers (Microsoft, Amazon, Alphabet, Meta, Oracle, CoreWeave) will deploy over $760B of capex in 2026, up from $410B in 2025 — but margin capture is concentrated in five chokepoints: chip design (Nvidia, 75% gross margin), foundry (TSMC, 66%), lithography (ASML, 53%), memory (SK Hynix/Micron, 70–85%), and networking (Arista/Broadcom, 62–77%).

• Assembly and rental — Dell (11% op. margin), Supermicro (8% gross), CoreWeave (6% op. margin) — capture almost none of it despite growing revenue 106–346% year-over-year.

• HBM memory’s 70–85% margins are real but cyclical, not structural — a formal scoring framework below puts it at 10/15 on durability, versus 13–15/15 for Nvidia, TSMC, and ASML. DRAM contract prices are up ~820% year-over-year into 2026, the steepest spike in at least three prior four-year memory cycles, and NAND spot prices have already fallen below contract prices — a classic early crack that has historically preceded a cycle turn.

• CoreWeave’s capex-to-revenue ratio exceeds 250%, financed substantially with debt — a materially different risk profile than hyperscaler capex funded by 37–46% operating margins on an already-profitable core business.

• Inference cost per token has fallen ~99.7% since 2023, yet hyperscaler capex is up 77% in 2026 alone — the DeepSeek shock of January 2025 (Nvidia’s largest one-day market-cap loss in history) tested the “efficiency kills demand” thesis in real time and it failed; Jevons paradox, not substitution, has governed this cycle so far.

• Verdict: overweight the durable-moat tier (Nvidia, TSMC, ASML, Arista, hyperscale cloud); trade, don’t hold, HBM memory; avoid or underwrite carefully the commodity-assembly and neocloud tier.

Thesis

Roughly $760 billion of hyperscaler and neocloud capital expenditure will move through the AI infrastructure stack in 2026 — Microsoft, Amazon, Alphabet, Meta, Oracle, and CoreWeave alone. Add Stargate, sovereign AI programs, and enterprise on-prem buildouts, and the gross figure clears $1 trillion. Almost none of this is evenly distributed. It concentrates in a handful of chokepoints — one lithography company, one foundry, three memory makers, two networking suppliers, one cooling specialist — and it evaporates almost entirely by the time it reaches the companies bolting the boxes together and renting out the racks.

The closest historical analogue is the late-1990s fiber-optic buildout: enormous, demand-justified capital investment in physical infrastructure, a handful of equipment suppliers who captured durable value (Cisco), and a much larger population of network operators who overbuilt capacity, took on debt to do it, and were largely wiped out or absorbed when the financing cycle turned — even though the underlying thesis (the internet would need vastly more bandwidth) was completely correct. The lesson from that cycle was never “don’t invest in the buildout.” It was “know which layer of the buildout you own.” The cloud capex cycle of 2010–2020 offers the more benign counter-example: AWS, Azure, and Google Cloud absorbed a decade of heavy infrastructure capex and came out the other side as some of the highest-margin software franchises in history, because the demand side (enterprise workload migration) proved durable enough, and the supply side concentrated enough, that scale itself became the moat. This piece is, in effect, an attempt to sort every company in the current AI buildout into one bucket or the other before the cycle answers the question for us.

This is the central fact of the AI buildout that market-map slides obscure: value capture in this stack is not correlated with dollars flowing through a layer — it’s correlated with how replicable that layer is. Nvidia converts 75 cents of every revenue dollar into gross profit. Dell converts eleven. Both are “AI infrastructure companies.” They are not remotely the same business.

The physical constraint underneath all of this is increasingly power, not capital. U.S. data center power demand is projected to climb from 31 gigawatts in 2025 to 41 GW in 2026 and 66 GW in 2027 — more than doubling in two years — while global data center power demand rises 27% in 2026 alone, from 104 GW to 132 GW, en route to a projected 290 GW by 2030. U.S. data centers’ share of peak summer electricity demand is set to nearly double, from 4.1% in 2025 to 8.5% by 2027. Every company in this piece is, at some level, racing against a grid buildout that moves on a slower clock than capital markets do — which is precisely why Vertiv’s backlog and the broader power/cooling layer matter more than their market cap alone suggests.

The rest of this piece works through the stack layer by layer, with the 2026 numbers, to build a durable mental model for where profit pools sit today and which of them survive the next three to five years of capacity build.

The Stack by Market Value

Before getting into margin structure, it’s worth seeing the stack the way public markets currently price it. The dispersion is extreme: Nvidia alone is worth more than the bottom eleven companies in this piece combined.

Three things stand out. First, the design and foundry layers (Nvidia, TSMC) and the hyperscale cloud layer (Microsoft, Amazon) account for the overwhelming majority of aggregate value — over $12 trillion combined among just four companies. Second, the memory names (Micron, Samsung, SK Hynix) have re-rated dramatically over the past twelve months on the back of the HBM cycle, with Micron alone now larger than ASML despite a much shorter history of sustained profitability. Third, and most tellingly for anyone doing bottom-up diligence in this sector, the assembly and neocloud tier — Dell, Supermicro, CoreWeave, Crusoe — trades at a small fraction of the value of the layers directly above and below it in the physical supply chain, despite processing enormous and growing volumes of capital. The market has already priced the framework this piece is describing; the question is whether it has priced it correctly for what comes next.

How Big Does This Get

Sizing estimates for the buildout vary by scope but agree on direction and magnitude. Goldman Sachs’ baseline model implies $765 billion of annual AI capex in 2026, scaling to $1.6 trillion annually by 2031. McKinsey puts total data center capital requirements at $6.7 trillion through 2030, of which $5.2 trillion is AI-specific — roughly 70% of all data center capacity built over the rest of this decade will be AI-driven. Morgan Stanley’s more conservative frame estimates nearly $3 trillion of AI infrastructure investment flowing through the global economy by 2028, with more than 80% of that spending still ahead of it, and has separately floated a $40 trillion figure for the total economic value AI diffusion could eventually create — a number that dwarfs the infrastructure capex required to enable it by roughly an order of magnitude.

What matters isn’t which estimate is right. It’s that every credible forecast has AI infrastructure spending compounding through at least 2030, off a base that has already tripled in three years. That is the demand backdrop against which every margin and moat argument in this piece has to be read. The layers with pricing power aren’t betting on a spending peak arriving soon — they’re pricing a multi-year runway, and the sell-side keeps revising its own numbers up, not down.

The Efficiency Paradox

The best objection to this whole piece isn’t about any one company. It’s macro: if AI intelligence is getting radically cheaper, why does infrastructure spending keep accelerating? That question already got tested in the real world, so here’s what actually happened.

In January 2025, Chinese lab DeepSeek released a model claiming frontier-class performance trained for a reported $5.6 million — a figure that, whatever its precision, was low enough to detonate the market’s assumptions about compute intensity. Nvidia lost $589 billion of market capitalization in a single trading day, the largest one-day value destruction of any company in history, on the fear that “great angst as to the impact for compute demand” (Cantor’s phrasing at the time) meant peak GPU spending had arrived early. Eighteen months later, Nvidia’s market cap sits at $4.85 trillion — more than 8x its post-DeepSeek trough — and combined Big Four hyperscaler capex has gone from $410 billion (2025) to a guided $725 billion (2026), a 77% increase in the same window the market once feared would mark the top.

The token-cost side of that chart is dramatic on its own terms. a16z’s “LLMflation” analysis and industry API pricing both show GPT-4-class inference falling from roughly $30 per million tokens at launch (March 2023) to under $0.10 per million tokens for equivalent capability by 2026 — two orders of magnitude in three years, faster than PC compute or dot-com-era bandwidth cost declines historically ran. That’s one of the fastest unit-cost collapses of any general-purpose technology on record. Capex rose anyway, at an accelerating rate, in the same window.

The answer is Jevons paradox, playing out about as cleanly as it ever has: when the cost of something useful falls, people use so much more of it that total spending rises even as the unit economics collapse. Morgan Stanley Research puts a precise shape on how this plays out over the rest of the decade: the AI infrastructure TAM grows from roughly $315 billion in 2025 to nearly $1 trillion by 2030, and within that growth, inference compounds at a 42% CAGR between 2025 and 2030 versus a materially slower rate for training — enough to flip the mix from training-majority today (roughly 75% training / 25% inference in 2025) to inference-majority

by the end of the decade (56% inference / 44% training in 2030).

Training doesn’t shrink in absolute terms — it keeps growing, from $236 billion to a projected $429 billion — it simply stops being the terminal use case. Inference, priced per query at a small and falling fraction of its 2023 cost, becomes the larger and faster-compounding half of the market anyway, because there are vastly more queries than there are training runs, and grows nearly seven-fold ($79 billion to $546 billion) over the same window. That shift favors the durable-moat and hyperscale-cloud tiers over a pure training-compute thesis, because inference workloads are more distributed, more latency-sensitive, and — importantly for Broadcom’s and the custom-silicon cohort’s argument — more receptive to purpose-built, less flexible, cheaper-per-query silicon than training workloads ever were. The efficiency curve is real and it’s fast, and so far it has accelerated demand rather than killed it, for every layer in this stack except the ones already flagged as weak below.

Scarcity Sets the Margin, Not the Spend

Every layer in this stack is growing. Growth isn’t what separates the winners — margin structure is. The pattern repeats at every node of the chain.

Layers with structural scarcity — a single supplier, a multi-year lead time to add capacity, or an entrenched software/IP moat — hold gross margins of 60–85% almost regardless of end-demand volatility. Layers that are capital-absorbing but replicable — assembly, integration, undifferentiated compute rental — get bid down to single-digit or low-teens margins the moment more than one credible competitor shows up with a checkbook.

Nvidia (chip design), TSMC (advanced foundry), ASML (EUV lithography), and the HBM oligopoly (SK Hynix, Samsung, Micron) all sit in the first category. Dell, Supermicro, and the neoclouds (CoreWeave, Crusoe) sit in the second. Networking (Arista, Broadcom) and power/cooling (Vertiv) sit in a durable middle tier: real engineering moats and real pricing power, but smaller absolute scale than the chip or foundry layers.

That spread is the real story. A 12-point gap between the top of the stack (77% gross margin at Broadcom’s custom silicon business) and the bottom (6% operating margin at CoreWeave, 8% gross margin at Supermicro) isn’t noise. It’s the market pricing scarcity correctly, layer by layer, in real time.

To make that judgment reproducible rather than impressionistic, score every layer on the three factors that actually determine whether a margin survives a capacity cycle: supply scarcity (how many credible suppliers exist today), switching cost / IP moat (what it costs a customer to leave), and capital intensity as a barrier (whether the capex required to enter is itself a deterrent, or merely a checkbook problem for a well-funded entrant). Score each 1 (weak) to 5 (strong) and the ranking falls out of the arithmetic rather than the narrative.

Two results cut against the size-based intuition of a market-cap ranking. First, HBM memory scores a 10/15 — respectable, but nearly three points below Nvidia and five below ASML — because its switching-cost score is only a 2: HBM is a qualified, multi-sourced commodity component, and a hyperscaler buying HBM3E today is not locked into SK Hynix the way it is locked into CUDA. Second, hyperscale cloud outscores Nvidia’s design layer on switching cost and capital intensity (5 and 5, versus Nvidia’s 5 and 3), a reminder that Nvidia’s moat is almost entirely a scarcity-and-software story, while AWS and Azure’s moat is scarcity-light but capital- and switching-cost-heavy. Both produce durable margin, but through different mechanisms — which matters for how fast each moat could erode, and under what conditions.

Layer 1 — Design

Nvidia earns roughly 75% gross margin. Dell earns 11% operating margin on its AI servers. Every allocator eventually asks why, and the answer is structural, not incidental.

Nvidia reported Q1 FY2027 (quarter ended April 2026) revenue of $81.6 billion, up 85% year-over-year, with data center revenue of $75.2 billion, up 92%. GAAP gross margin was 74.9%. Operating income was $53.5 billion — a 65.6% operating margin — and net income was $58.3 billion, up 211% year-over-year, on free cash flow of $48.6 billion. The company’s market capitalization sits at roughly $4.85 trillion, making it the most valuable company on earth.

Nvidia doesn’t manufacture a single wafer. It designs silicon and owns CUDA, the software layer that has absorbed fifteen years of developer tooling, libraries, and institutional muscle memory. That software moat is what lets Nvidia charge a price for the H100/B200/Rubin generations that reflects the value of the entire AI training and inference workflow, not the marginal cost of a chip. Demand exceeds supply at every node TSMC can produce; Nvidia is the scarce resource, and it prices like one.

This scarcity cuts both ways. Nvidia’s own disclosures show four direct customers each representing more than 10% of total revenue in Q3 FY2026 — 22%, 15%, 13%, and 11% respectively, a combined 61% of revenue from four unnamed hyperscaler-scale buyers. That concentration is the natural consequence of selling the scarcest input in the stack to the handful of companies large enough to buy it at scale. But it also means Nvidia’s growth trajectory is now more sensitive to the capital-allocation decisions of four boardrooms than diversified revenue bases usually are — the same four boardrooms, in some combination of Microsoft, Amazon, Meta, and Google, that are simultaneously funding the custom-silicon programs built to reduce that dependency over time.

Dell’s Infrastructure Solutions Group (ISG), by contrast, posted a record $29 billion in Q1 FY2027 revenue with AI-optimized server revenue of $16.1 billion — up 291.7% year-over-year — against an ISG operating margin of just 10.5%, itself up only 80 basis points despite AI server volume growing nearly 800%. Dell closed FY2026 with $64.1 billion in cumulative AI orders and a $43–51 billion AI backlog. The volume is real and enormous. The margin is not moving.

The mechanism is simple: Dell (and Supermicro, whose gross margin has run between 6.3% and 10.1% across recent quarters despite $38.9–40.4 billion in FY2026 revenue guidance) buy the same Nvidia GPU, the same SK Hynix HBM, the same Broadcom NIC that every other integrator can buy, and assemble them into a box. There is no proprietary IP in the assembly step that a customer can’t get from three other vendors within a quarter. Dell and Supermicro are price-takers on their most expensive input (the GPU) and price-takers on their output (the finished server) — squeezed from both sides in a business where the customer, a hyperscaler or enterprise, can and does multi-source.

It’s the cleanest illustration in this whole framework: identical end demand, identical growth rate, radically different margin, because one layer is a monopoly-adjacent software platform and the other is a commodity assembly line.

The clearest way to see this is to follow a single chip from the fab to the rack. Epoch AI’s cost-breakdown analysis puts total manufacturing cost (COGS) for one Blackwell B200 GPU at roughly $6,400 — and within that figure, HBM memory alone is 45% ($2,900), ahead of the logic die itself (22%, $1,400), CoWoS advanced packaging (17%, $1,100), and substrate/testing (16%, $1,000). Nvidia then sells that same GPU for a street price in the $30,000–$40,000 range: a five-to-six-times markup captured entirely at the design layer, on top of a component cost structure in which the memory supplier has already embedded its own 70%+ gross margin before the chip is even assembled.

Scale that to a full GB200 NVL72 rack — 72 of these GPUs plus Grace CPUs, NVLink switching, and cooling infrastructure — and the system-level price runs to roughly $3.1 million for the compute portion, $3.9 million all-in once networking and storage are included. Dell or Supermicro’s job is to take that $3.9 million of mostly Nvidia-priced, SK-Hynix-priced, and Broadcom-priced components, add chassis, cabling, integration labor, and a support contract, and mark the whole thing up by single digits. Every dollar of margin in that rack was captured before the box was assembled. Assembly is where margin goes to be spent, not made.

The Layer Beneath the Layer

Nvidia’s 75% gross margin is a hardware number attached to a software moat, and the two are worth separating, because software is the layer in this stack most resistant to every competitive pressure described elsewhere here. CUDA isn’t one product. It’s roughly two decades of accumulated libraries, compilers, and developer tooling (cuDNN, NCCL, TensorRT, and hundreds of downstream frameworks) that make Nvidia silicon the default target for every new model architecture before anything else gets considered. That accumulated software surface is precisely what custom silicon programs at Google, Amazon, and Microsoft have to replicate from a standing start, and why, even with $225 billion of committed Trainium spend, Amazon’s own leadership frames the custom-silicon push as a workload-specific complement to Nvidia dependency rather than outright replacement on any near-term timeline. Beneath CUDA sits a second software layer that gets even less investor attention: the orchestration, scheduling, and cluster-management tooling (Slurm, Kubernetes-based GPU schedulers, Ray) that determines whether a given cluster actually achieves the utilization its hardware spec implies. A cluster running at 55% effective utilization instead of 85% is, economically, a cluster with roughly a third of its capital sitting idle, which makes this orchestration layer, though it captures little margin directly today, one of the more interesting adjacent software opportunities the current buildout is creating.

The strongest bear case on the CUDA moat is worth taking seriously, because it’s the biggest swing factor in whether Nvidia’s design-layer score stays a 13/15 or drifts toward Broadcom’s contested 9/15 over a five-year horizon. PyTorch, the framework most model training now runs through, has spent several years deliberately abstracting away direct CUDA calls behind its own compiler stack (torch.compile, TorchInductor); OpenAI’s Triton language does the same for custom kernel authorship, letting a developer write hardware-agnostic code that can, in principle, target AMD’s ROCm or a custom ASIC’s compiler backend with far less rewrite effort than five years ago. If that abstraction layer keeps maturing, the actual switching cost of leaving Nvidia silicon falls even as CUDA itself stays dominant, because the thing developers depend on stops being CUDA specifically and starts being the compiler layer sitting on top of it. The counter-evidence: this transition has been “underway” for essentially the entire period Nvidia’s data center revenue compounded from single-digit billions to $75 billion a quarter, and every attempted software abstraction to date has still shipped its best-optimized path on Nvidia hardware first. The moat looks real today and slowly eroding at the edges — not collapsing, not permanent — which is why the framework scores it a 5 on switching cost while flagging it as the dimension most likely to move.

Layer 2 — Foundry

TSMC sits at the foundry chokepoint of the stack, and the bottleneck inside it has a name: CoWoS.

TSMC posted Q1 2026 revenue of $35.9 billion, up 40.6% year-over-year, at a record 66.2% gross margin, 58.1% operating margin, and 50.5% net margin. Full-year 2026 revenue is guided to grow more than 30% in USD terms. Market capitalization: roughly $2.1 trillion.

TSMC’s moat isn’t really “advanced lithography” — that’s ASML’s layer, one step upstream. TSMC’s moat is the accumulated manufacturing yield, process know-how, and packaging capacity that no other foundry has matched at volume. The specific chokepoint that determines how much AI silicon can physically reach the market in any given quarter is CoWoS (chip-on-wafer-on-substrate), the advanced packaging process that stitches GPU compute dies to HBM stacks.

CoWoS capacity has gone from roughly 35,000 wafers per month at the end of 2024 to about 75,000 at the end of 2025, with a target of 125,000–130,000 by the end of 2026 — nearly a 4x expansion in two years. TSMC’s own CEO described the line as “sold out through 2026” as of the company’s June 2026 annual meeting. It’s the most important supply constraint in the entire AI stack: it doesn’t matter how many GPUs Nvidia can design or how much HBM SK Hynix can fab if CoWoS capacity can’t marry the two together. TSMC captures margin here not because it competes on price, but because for the 3-nanometer/2-nanometer AI accelerator category, there is effectively no alternative supplier at scale. Samsung Foundry and Intel Foundry remain multiple process generations and multiple years of qualification behind on both logic and advanced packaging.

The Layer Investors Overlook: Optics

Every GPU cluster large enough to train a frontier model is, physically, an interconnect problem before it’s a compute problem. Copper reaches its practical bandwidth-distance limit within a single rack; everything beyond that — GPU-to-GPU across racks, rack-to-rack across a data hall — runs over optical transceivers. As cluster sizes have scaled from thousands to hundreds of thousands of GPUs, the optics bill of materials per cluster has scaled with it: a single large training cluster today consumes hundreds of thousands of 800G and increasingly 1.6T optical transceivers, a cost line that barely existed at this scale three years ago. This is a growing profit pool, occupied by transceiver and silicon-photonics suppliers sitting adjacent to Broadcom’s and Arista’s networking businesses, and it’s also a genuine physical bottleneck: optics manufacturing capacity, like CoWoS, cannot be added instantly, and a shortage here caps effective cluster size regardless of how many GPUs and how much HBM are sitting in inventory. It’s the least-discussed chokepoint in this entire stack relative to how much it actually matters.

Layer 3 — Lithography

One step further upstream sits ASML, the Dutch company that is the sole global supplier of EUV (extreme ultraviolet) lithography systems — the machines TSMC, Samsung, and Intel all need to print anything below the 7nm node. ASML’s Q1 2026 net sales were €8.8 billion at a 53% gross margin; full-year 2026 guidance calls for €34–39 billion in revenue (roughly 11.6% growth at the midpoint) with 51–53% gross margin. Backlog stood at €38.8 billion at year-end 2025 after a record €13.2 billion bookings quarter. Market capitalization: approximately €600B / $690 billion.

ASML’s margin is the lowest of the “scarcity” tier in this piece, not because its moat is weaker, but because its business model runs on multi-year order books and long manufacturing lead times rather than the vertiginous spot-market dynamics of GPUs or memory. It is, structurally, the most durable monopoly in the entire chain: there is no second EUV supplier, full stop, and there won’t be one for the foreseeable future given the two-decade, multi-hundred-billion-dollar R&D investment embedded in the current machines. ASML’s growth rate looks modest next to Nvidia’s or Micron’s because it doesn’t need to chase the AI cycle. Every advanced chip on earth, AI or otherwise, already runs through its tools.

Layer 4 — Memory

High-bandwidth memory (HBM), the memory stacked directly onto the GPU package to feed it data at the bandwidth AI training requires, has been the best-performing sub-sector in semiconductors in 2026, and it deserves the most caution of anything in this piece.

SK Hynix dominates the category, with roughly 50–56% global HBM share, reported Q1 2026 revenue of approximately ₩52.6 trillion (~$35.5 billion), up 198% year-over-year, at an operating margin near 72%. Its market cap has roughly doubled over the trailing twelve months to around $586 billion. Micron has ridden the same wave from the number-three HBM position (roughly 20–24% share): fiscal Q3 2026 revenue hit $41.46 billion, up 346% year-over-year, at an 84.9% non-GAAP gross margin, with FQ4 guidance of $50 billion in revenue at roughly 86% gross margin — a business that traded at single-digit multiples on commodity-cycle earnings as recently as 2023 now carries a $1.3 trillion market cap. Samsung holds the remaining roughly 28% and is racing to qualify HBM4 for the next Nvidia platform generation.

Samsung is the wild card inside this layer, because it’s the only company in the entire stack playing on two boards at once: it holds roughly 28% of HBM share directly behind SK Hynix and Micron, and it’s simultaneously the only credible, if distant, alternative to TSMC in advanced-node logic foundry. That dual exposure makes Samsung the best-positioned incumbent to benefit if either the memory oligopoly holds or TSMC’s packaging bottleneck creates enough overflow demand for a qualified second source, and the most likely company to absorb margin if HBM pricing normalizes before Samsung’s foundry business has scaled enough to offset it. At a $1.34 trillion market cap, Samsung trades as a memory story today; the foundry option value is, for now, priced closer to zero.

The margins here, 70–85% gross, look identical to Nvidia’s on paper. They aren’t the same kind of margin. Nvidia’s margin is protected by a software ecosystem that took fifteen years to build and that customers cannot easily leave. HBM’s margin is protected by a supply shortage: three suppliers, all racing to add capacity, in a category that has been through boom-bust cycles roughly every four years since DRAM existed as a product. Peak-to-trough swings recur on an almost metronomic four-year cadence, with prior cyclical turns clustering around 2010–2014, 2014–2018, and 2018–2022. SK Hynix, Samsung, and Micron are all expanding HBM capacity aggressively in response to today’s pricing, and capacity additions of this kind, in memory, have never once failed to eventually overshoot demand.

What makes the current cycle different is its slope, not its shape. Morgan Stanley Research and TrendForce data show DRAM contract pricing up roughly 820% year-over-year heading into 2026 — a spike that dwarfs any prior peak in at least three previous cycles, and the kind of move that historically has marked a cycle’s most extended phase rather than its middle innings.

Two additional signals from the same data cut in different directions. First, supply-chain inventory — measured in weeks of stock held across DRAM/NAND suppliers, PC OEMs, module makers, server makers, and smartphone makers — peaked in mid-2024 to early 2025 and has been normalizing through 2026, which removes the destocking buffer that cushioned prior cycles and means fresh demand or supply news now hits pricing more directly. Second, and more tellingly, spot and contract prices have decoupled in opposite directions across the two memory types. DDR5 spot pricing (~$45) currently runs ahead of contract pricing (~$37.50), consistent with a market still tightening in real time, while 512GB NAND/SSD spot pricing (~$84) has fallen below contract pricing (~$118) — an inversion that typically shows up when channel buyers start pulling back faster than the long-lead-time OEM contracts they’re locked into can adjust. That NAND inversion isn’t proof the cycle has turned. It’s the kind of small, early crack in the data that historically precedes one, and it’s the clearest reason the 2028 mean-reversion call earlier in this piece is a base case rather than a low-confidence guess. The 2026 HBM margin structure is real cash flow today and a legitimate multi-year window, but it’s a cyclical profit pool wearing the appearance of a structural one, and it’s the layer in this piece most likely to see today’s 70%+ margins compressed first.

Layer 5 — Networking and Custom Silicon

Broadcom occupies an unusual dual position. Its legacy networking/semiconductor business carries company-wide gross margins around 77% (down slightly from 79.1% a year earlier as mix shifts). Its faster-growing custom AI silicon (XPU) business, designing application-specific accelerators for hyperscalers who want Nvidia-alternative compute, generated $8.4 billion in AI revenue in Q1 FY2026, up 106% year-over-year, against a $73 billion AI backlog and a company target of $100 billion in annual AI semiconductor revenue by 2027. Market capitalization has moved between roughly $1.8 and $2.3 trillion through 2026 depending on the week, among the most volatile large-cap valuations in the stack, reflecting genuine investor uncertainty about how much of Nvidia’s design-layer margin Broadcom’s custom-ASIC model can ultimately capture.

Arista Networks is the purer comparison for the “networking has real pricing power” argument: FY2025 revenue of $9.0 billion (up 28.6%), FY2026 guidance raised to roughly $11.5 billion (27.7% growth), and gross margin held steady at 62–64% even as AI-cluster interconnect demand strains the entire supply chain. Arista’s moat is software-defined switching (EOS) plus deep incumbency in the largest cloud and AI cluster deployments, a smaller-scale, higher-durability cousin of Nvidia’s CUDA lock-in, at a $220 billion market cap.

Custom silicon is the most important competitive threat to Nvidia’s margin structure in the entire stack, and Broadcom is the company best positioned to monetize that threat, which is exactly why its market cap has been so volatile. It’s simultaneously a bet for the Nvidia alternative and against Nvidia’s durability.

That threat is no longer speculative. Custom AI ASIC shipments are projected to grow 44.6% in 2026, nearly three times the 16.1% growth rate projected for merchant GPUs, as Google’s Ironwood TPU, Amazon’s Trainium 3, Microsoft’s Maia 200, and Meta’s MTIA all ship in volume. Amazon has disclosed more than $225 billion in Trainium revenue commitments, and its custom-silicon program reportedly crossed a $20 billion annualized run rate in Q1 2026 (largely internal transfer pricing to AWS rather than external merchant sales, but real capital allocation nonetheless). Broadcom and Marvell together control an estimated 95% of the custom-ASIC co-design market, which means the “threat to Nvidia” and “the next leg of Broadcom’s growth” are, functionally, the same trade.

None of this displaces Nvidia in the near term. CUDA’s ecosystem lock-in means workload migration to custom silicon happens model generation by model generation, not quarter by quarter. But it caps the multiple the market should be willing to assign to the durability of Nvidia’s 75% margin, even as the level of that margin stays intact for the foreseeable future.

Layer 6 — Power and Cooling

Vertiv is the cleanest “picks and shovels” business in the stack: data center power and thermal management, benefiting from the industry’s forced pivot to liquid cooling as rack power densities go vertical. The industry-average rack draws roughly 7.6kW. Nvidia’s GB200 NVL72 draws 120–140kW — 16 to 18 times the legacy average — and the 2026 Vera Rubin generation (VR200 NVL72) is specced at 190–230kW, with the long-context CPX variant near 370kW. The 2027 Rubin Ultra (NVL576) roadmap points to roughly 600kW per rack, which is why Nvidia is simultaneously pushing the industry toward 800-volt DC facility power architecture just to make racks north of 600kW physically feasible. Air cooling can’t dissipate any of this. Liquid cooling isn’t an option at these densities; it’s a requirement, and that requirement is what turns Vertiv’s backlog into multi-year, engineering-locked revenue rather than a commodity hardware order. Q1 2026 revenue was $2.65 billion (up 30.1% year-over-year); full-year guidance was raised to $13.5–14.0 billion; adjusted operating margin expanded 430 basis points to 20.8%; backlog more than doubled to over $15 billion, covering 12–18 months of forward revenue. Market cap: roughly $121 billion.

Vertiv’s margin sits well below the chip layer but meaningfully above the assembly layer, for a specific reason: power and cooling infrastructure is engineered and specified, not commoditized. It requires deep integration with a data center’s electrical and mechanical design, creates real switching costs once installed, and has a backlog structure that gives multi-quarter revenue visibility other layers in the stack lack. It’s a smaller moat than the chip layer, but a durable one — a fact the market underappreciated through 2023–2024 and has since substantially repriced.

Layer 7 — Cloud and Neocloud

Hyperscale cloud and neocloud look like the same business — renting AI compute — and this is where the framework produces its sharpest, most investable distinction: they are not, financially, the same business at all.

AWS posted Q1 2026 segment revenue of $37.6 billion (up 28% year-over-year) and segment operating income of $14.2 billion, a 37.7% operating margin. Microsoft Azure grew 39% year-over-year in constant currency in the same period, with company-wide operating margin at 46.3%, up from 45.7% a year prior, even as Microsoft guided to roughly $190 billion of calendar-2026 capex (up 61% from 2025). Oracle Cloud Infrastructure (OCI) grew 93% to $5.8 billion in its most recent quarter, backed by a staggering $638 billion remaining-performance-obligations backlog (up 363% year-over-year), but Oracle’s implied margin on this business is thin, and the stock gave back roughly a third of its market cap between early June and early July 2026 (from ~$614 billion to ~$450 billion) as investors reassessed how much of that backlog converts to profitable revenue versus GPU-lease pass-through.

CoreWeave, the largest pure-play “neocloud,” tells the cautionary version of this story in numbers: 2026 revenue guidance of $12–13 billion against a $99.4 billion contracted revenue backlog, 2026 capex guidance of $31–35 billion, roughly $25 billion of debt on the balance sheet as of Q1 2026, and an adjusted operating margin of only about 6%. Crusoe, still private, is scaling revenue toward an estimated $2 billion in 2026 and has seen its valuation nearly triple in eight months, from just over $10 billion (October 2025 Series E) to reported talks near $30 billion (July 2026) — a re-rating pace that outstrips even the memory names.

The mechanism behind the margin gap is depreciation and financing structure, not customer demand. Hyperscalers depreciate GPU capex over five-plus years against a diversified, high-margin software/services revenue base and can subsidize AI infrastructure economics with the rest of the business. Neoclouds are frequently financing GPU purchases with debt collateralized by the GPUs themselves, running at far higher balance-sheet leverage, and monetizing a single asset class (raw compute rental) into a market where hyperscalers are simultaneously their landlord, their supplier, and increasingly their competitor.

CoreWeave’s $99 billion backlog is the bull case for the neocloud model: real, signed, multi-year demand from credible counterparties, largely Microsoft, OpenAI, and other frontier labs. Its $25 billion of debt against a 6% operating margin is the bear case: neoclouds have taken on hyperscaler-level capital intensity without hyperscaler-level margin protection, and CoreWeave’s market cap has fallen from prior highs to roughly $47–58 billion through June 2026 as the market repriced that mismatch.

Growth Is Universal; Margin Is Not

Isolate growth alone and every layer in this stack looks like a buy. That’s exactly the trap.

Dell’s AI server business (+292%) and Micron’s memory business (+346%) are growing faster than Nvidia (+85%) or TSMC (+41%). By a naive growth-rate screen, the assembly and memory layers look like better businesses than the design and foundry layers. They aren’t, because growth in this stack is a function of end-market demand, which every layer shares roughly equally, while margin is a function of competitive structure, which varies enormously by layer. A commodity business growing 292% is still a commodity business — it simply commoditizes a larger dollar volume every quarter. It’s the most common mistake in surface-level analysis of the AI infrastructure trade, and it’s why the margin-by-layer view earlier in this piece matters more than any growth chart on its own.

Capital Intensity

The other variable growth-rate screens miss is how each layer’s expansion is financed, and this is where the durable and temporary tiers diverge most sharply in risk, not just in margin.

TSMC and ASML both run enormous capital expenditure programs — TSMC’s annual capex runs in the $40–52 billion range — but they fund it from operating cash flow generated at 50%+ net margins, leaving the balance sheet essentially unlevered relative to earnings power. Hyperscalers are similar in kind if not in scale: Microsoft’s ~$190 billion 2026 capex plan and Amazon’s roughly $200 billion figure are both funded by companies with 37–46% operating margins on existing, diversified, high-margin revenue bases. AI capex is large in absolute terms but small relative to the cash-generating power of the businesses funding it.

CoreWeave is the inverse case, and the clearest illustration of why capital intensity is a risk factor independent of growth or even margin. Against 2026 revenue guidance of $12–13 billion, the company is guiding to $31–35 billion of capex, a ratio north of 250%, financed substantially with debt (roughly $25 billion outstanding as of Q1 2026) collateralized by the GPUs the capex buys. That’s a fundamentally different risk profile than a hyperscaler spending a similar dollar amount out of a profitable core business: CoreWeave’s downside case involves creditors, not just a lower stock multiple, if its $99.4 billion revenue backlog fails to convert to cash flow on schedule. Oracle’s OCI buildout sits somewhere in between: funded by a large, profitable database and applications franchise, but increasingly reliant on customer prepayments to bridge the gap between capex outlay and recognized revenue, a structure that inflated Oracle’s own reported capex by $20–25 billion in FY2026 and that the market punished with a roughly 27% pullback in market cap between early June and early July 2026 alone.

The rule of thumb for allocators: the safest capital intensity in this stack is capital intensity funded by an already-profitable, already-diversified business. The riskiest is capital intensity funded by debt against a single, undifferentiated, cyclically exposed asset class. That’s the CoreWeave and Crusoe model, and it’s why both companies’ market caps and funding structures deserve more scrutiny than their revenue growth rates alone would suggest.

Putting It Together: Margin vs. Growth Across the Whole Stack

Pricing power (upper right) — Nvidia, Micron, SK Hynix, Broadcom’s AI business. High growth and high margin. The risk here is duration: for the memory names in particular, today’s margin is a function of a supply shortage, not a permanent structural advantage.

Scarce and slow (upper left) — ASML, TSMC, Arista, hyperscale cloud. High margin, comparatively modest growth. These are the most durable profit pools in the stack precisely because they aren’t chasing the AI cycle; AI demand is additive to an already-entrenched, already-profitable core business.

Commodity growth (lower right) — Dell ISG. Enormous volume growth, thin and barely-improving margin. Clear evidence that revenue growth and value capture have become decoupled at the assembly layer.

Structurally squeezed (lower left) — Supermicro, CoreWeave, Oracle OCI (on an implied-margin basis). Real revenue, real demand, and a cost structure (component costs, GPU depreciation, debt service) that consumes nearly all of it.

Where the Money Is Actually Coming From

None of this margin analysis matters without sizing the capital flow behind it. Big Tech capex for AI infrastructure now runs at a scale that exceeds the GDP of most countries.

Microsoft, Amazon, Alphabet, and Meta combined are on pace for roughly $725 billion of 2026 capital expenditure, up from about $410 billion in 2025 — a 77% increase in a single year. Layer in Oracle’s roughly $50 billion (a figure that understates cash capex by $20–25 billion once customer GPU prepayments are excluded) and CoreWeave’s $31–35 billion, and six buyers alone deploy over $760 billion into this stack in 2026. Consensus sell-side estimates for total hyperscaler AI capex in 2026 sit at or above $500–527 billion even before neoclouds and sovereign programs (Stargate’s initial $100 billion tranche of a targeted $500 billion by 2029) get added in.

That’s the scale of demand pull that explains why every layer above the assembly tier is capacity-constrained rather than demand-constrained. The binding question for every company in this piece isn’t “will demand arrive” — it already has, several times over, in the form of signed backlog. The binding question is which layers can convert that demand into margin that survives the next capacity cycle.

Where Competition Is Intensifying — and Where It Isn’t

Four fronts to watch, ranked by how much they threaten today’s margin structure:

Custom silicon vs. Nvidia (highest threat to the top of the stack). Google’s TPU program, Amazon’s Trainium, Meta’s MTIA, and the broader Broadcom/Marvell custom-ASIC design business represent the only credible long-term pressure on Nvidia’s 75% gross margin. Broadcom’s $73 billion AI backlog and $100 billion 2027 target are the clearest market signal that hyperscalers are willing to spend real design dollars to reduce Nvidia dependency — not because CUDA can be replicated quickly, but because at hyperscaler volume, even a partial workload shift to purpose-built silicon changes the unit economics enough to justify the multi-year design investment.

Memory capacity response (highest threat to the current HBM supercycle). SK Hynix, Samsung, and Micron are all expanding HBM capacity in direct response to 2025–2026 pricing, and all three have publicly committed to next-generation HBM4 qualification timelines that pull forward the point at which combined supply catches demand. Every previous memory upcycle in the history of the DRAM industry has ended the same way, with the suppliers’ own capacity response overshooting demand and collapsing pricing, typically within 18–30 months of the cycle’s peak signing activity. Nothing suggests this cycle breaks that pattern structurally; there is only evidence that AI training and inference token volumes are growing fast enough to have kept HBM undersupplied for longer than earlier cycles, and that the qualification complexity of HBM4 (versus commodity DDR) may modestly extend the time it takes new capacity to actually count as usable supply. Directionally, though, three well-capitalized competitors racing to add capacity into a 70%+ gross margin is about as reliable a mean-reversion setup as semiconductors produce.

Neocloud commoditization (highest threat to the bottom of the cloud layer). CoreWeave, Crusoe, Lambda, and a growing list of GPU-rental businesses are underwritten by the same asset (Nvidia GPUs) financed against the same buyer pool (frontier AI labs and hyperscaler overflow demand). As hyperscalers build out their own capacity and become less reliant on neocloud overflow, and as more entrants chase the same financing structure, this layer faces the fastest margin compression risk of anything in the stack outside of memory.

Foundry and packaging alternatives (lowest near-term threat, highest long-term uncertainty). Samsung Foundry and Intel Foundry are both investing to close the gap with TSMC on both leading-edge logic and advanced packaging, and both are multiple years and multiple process nodes behind at AI-relevant volume. Geopolitical risk (Taiwan) is the more relevant near-term variable here than competitive risk. CoWoS capacity being “sold out through 2026” is a supply problem TSMC’s competitors would need years to meaningfully address even if they matched the technology today. A Taiwan-related supply disruption wouldn’t shift share to a competitor so much as it would remove 90%+ of the world’s leading-edge AI chip supply from the market simultaneously — a tail risk with no meaningful hedge anywhere else in this stack, and an argument that TSMC’s margin premium is still too low relative to the systemic risk it’s underwriting for the entire industry.

Moat Durability

This chart matters more than any other in this piece for allocators, because market capitalization is a poor proxy for moat durability across this stack. Broadcom is larger than Arista by market cap, but Arista’s switching-software moat is arguably more durable than Broadcom’s customer-co-design moat, which depends on hyperscalers continuing to prefer bespoke silicon partnerships over vertical integration. Micron is a larger company than Vertiv, but Vertiv’s engineered, backlog-protected power/cooling business is structurally more durable than Micron’s HBM margin, which is a function of a supply shortage rather than an unassailable technology lead.

The durable tier — ASML, TSMC, Nvidia, hyperscale cloud, Arista, Vertiv — shares a common feature: replicating any of them requires either a monopoly-grade technology position, a decade-plus of accumulated manufacturing yield, an entrenched software ecosystem, or contractually embedded switching costs. The temporary tier — HBM suppliers, Broadcom’s custom-ASIC business, and especially Dell/Supermicro and the neoclouds — shares the opposite feature: their current margin is a function of a supply/demand imbalance (memory, GPUs) or a temporary strategic preference (custom co-design), not a barrier competitors cannot eventually cross.

“Durable” doesn’t mean these companies’ growth rates stay elevated forever. ASML’s ~12% growth this year already shows that durable-moat businesses can grow slower than the cycle’s hottest names while still compounding capital more reliably. It means the rate of return on the next incremental dollar of capacity stays high, because a competitor can’t show up in eighteen months and bid it away. That distinction, durability of returns versus rate of growth, is the one growth-rate-only screens systematically miss. It’s also why this framework treats Broadcom’s custom-silicon business (real growth, contested durability) so differently from Arista’s switching business (slower growth, largely uncontested durability), despite both sitting in the same “networking-adjacent” bucket on a superficial market map.

Implications

For venture and growth investors: the layers worth funding are the ones where a new entrant can either attack an incumbent’s replicability weakness (better financing structures or differentiated workload specialization for neoclouds, alternative packaging/interconnect approaches that ease the CoWoS bottleneck, novel cooling architectures that outpace Vertiv’s engineering lead), or become the next layer’s version of CUDA: a software or orchestration moat that makes switching costly regardless of whether the underlying hardware is commoditized. Capital deployed into “another GPU cloud” or “another server integrator” without either of those two attributes goes into the least defensible part of the entire value chain, regardless of how fast the segment’s revenue is growing.

For public-market allocators: the framework argues for structural overweight to the durable-moat tier (ASML, TSMC, Nvidia, Arista, hyperscale cloud parents) funded by structural underweight to the commodity-assembly and neocloud tier (server OEMs, pure-play GPU rental), with HBM memory treated as a high-conviction trade, not a long-term holding. The margin is real and probably persists through 2027, but memory cycles do not skip a beat forever, and 70–85% gross margins on DRAM-derived products have never once proven permanent.

For corporate strategists inside hyperscalers and enterprises: the custom-silicon investment thesis is validated by Broadcom’s backlog growth, but the CUDA software moat means the payback period on internal silicon programs is measured in years of workload migration, not quarters, and the capex commitment (Big Four at $725 billion in 2026) suggests the industry is underwriting Nvidia’s near-term pricing power even while actively trying to escape it. Enterprises buying AI infrastructure directly rather than through a hyperscaler should weight the capital-intensity framework above as heavily as the headline price-per-GPU-hour: a neocloud offering below-market pricing today is very often financing that discount with leverage that carries more risk than it appears to, and a vendor’s balance sheet is now a legitimate infrastructure-procurement diligence item in a way it simply wasn’t three years ago.

On timeline: the durable-tier thesis (Nvidia, TSMC, ASML) is a multi-year holding period call with limited need for precise entry timing, given the underlying scarcity is structural rather than cyclical. The HBM memory trade has a shorter effective horizon, probably 12 to 24 months of further upside before capacity additions from all three suppliers begin to bite, based on the historical cadence of prior memory cycles. The neocloud and assembly-layer caution flagged throughout this piece is closer to an immediate positioning call than a multi-year thesis: the capital-structure risk in that tier is already visible in 2026 pricing (CoreWeave and Supermicro’s compressed market caps relative to revenue growth are the market’s own early verdict), not a risk that requires a future catalyst to materialize.

Summary

Value in the AI infrastructure stack accrues to scarcity, not spend. Nvidia, TSMC, and ASML sit at genuine chokepoints — design IP with a software moat, manufacturing yield with no substitute at volume, and a lithography monopoly with no second supplier — and they price accordingly, at 53–75% gross margins that have held up across multiple demand shocks. HBM memory suppliers currently sit alongside them on margin (70%+ gross) but not on durability: their profit pool is a supply-shortage artifact that history says will eventually mean-revert. Networking (Arista) and power/cooling (Vertiv) occupy a smaller but genuinely durable middle tier, protected by software lock-in and engineering/backlog moats respectively. Server assembly (Dell, Supermicro) and neocloud GPU rental (CoreWeave, Crusoe) sit at the bottom of the stack by design: undifferentiated, multi-sourced, and priced down to single-digit or low-teens margins regardless of how fast their revenue grows, with the neocloud cohort carrying the extra risk of hyperscaler-grade leverage without hyperscaler-grade margin protection.

The $760 billion-plus flowing through this system in 2026 will keep growing. Who keeps it won’t change nearly as much as the market map suggests, and the parts of the map most likely to change are exactly the parts, like HBM memory and neocloud rental, that look most attractive on a trailing-twelve-month growth chart today.

For a decade the AI narrative was a compute narrative. The scarce resource was FLOPS, the winner was whoever owned the accelerator, and the network was plumbing. That framing is now obsolete. In a modern AI factory, a single training job spans tens of thousands of GPUs that must behave as one machine, and the fraction of wall-clock time those GPUs spend waiting on each other — not computing — is set almost entirely by the network. The interconnect has become the second silicon.

This piece maps the whole stack — the physical and logical layers that turn a warehouse of accelerators into a coherent supercomputer — and reads it the way an investor should. The consensus that networking is “the new moat” is directionally right but mispriced. The durable value is concentrating in three places a GPU-centric analysis tends to miss: the software that converts rated bandwidth into delivered bandwidth, the upstream optical supply chain everyone is capacity-constrained on, and the reliability layer that decides whether a 100,000-GPU run finishes at all. Compute is a spec sheet; the network is an operational outcome. Start with the map.

Figure 1 · The three domains of AI networking. Scale-up binds a handful-to-a-thousand accelerators into one memory space at terabytes/second over copper; scale-out lashes thousands-to-millions of GPUs into a cluster over optical Ethernet or InfiniBand; scale-across stitches AI factories together across buildings and metros. Each domain has a different physics, a different incumbent, and a different margin profile.

The Framework: Three domains, three physics, three markets

The single most useful mental model for AI networking is that there is no such thing as “the network.” There are three, nested like Russian dolls, and they are governed by different constraints. Scale-up is the innermost: it fuses a small number of accelerators into a single, coherent memory fabric where any GPU can read another’s memory in roughly 100 nanoseconds. Scale-out is the cluster fabric that connects those tightly-coupled islands into a machine of tens or hundreds of thousands of GPUs. Scale-across is the newest layer, connecting entire AI factories across buildings and metros as single sites run into power ceilings.

The reason this taxonomy matters commercially is that bandwidth, latency, and reach trade off against each other and against cost. Copper is nearly free and draws almost no power, but it dies at about two meters. Optics reach kilometers but burn watts and dollars per bit. The entire architecture of an AI supercomputer is an exercise in keeping the highest-bandwidth traffic on the cheapest, shortest, coolest medium — and pushing to light only when distance forces the issue. Follow the money, and it maps almost perfectly onto that physics.

Why It Gates Scaling: The utilization gap is a networking problem

The reason the interconnect commands this share of spend is visible in a single metric: Model FLOPS Utilization, the fraction of a GPU’s theoretical throughput a real training run actually achieves. Frontier runs commonly sustain under 50% MFU — more than half of the silicon you paid for is idle at any instant — and communication is a primary culprit. As models shard across more GPUs and parallelism widens, the share of wall-clock time spent moving data rather than computing rises. Past a certain scale the network, not the accelerator, sets the ceiling. That is the whole reason scale-up bandwidth (NVLink) and lossless scale-out fabrics exist: to keep the compute fed.

The proof is operational, not theoretical. When xAI built Colossus — 100,000 H100s, since expanded toward 200,000 GPUs including GB200 — on NVIDIA Spectrum-X, the fabric sustained 95% delivered throughput with zero flow-collision packet loss. NVIDIA pegs standard Ethernet at roughly 60% in the same role. That 60%→95% is a ~1.6× swing in effective bandwidth from the fabric alone — the difference between a GPU that earns and a GPU that waits. At a cluster costing tens of billions, that gap is the single highest-return line item in the building.

Scale-Up · The Memory Fabric: NVLink, NVSwitch, and the copper moat

Scale-up is where NVIDIA is strongest and where the physics are most punishing. When a model is sharded across GPUs using tensor parallelism, those GPUs exchange activations on every layer of every forward and backward pass. That traffic is enormous, latency-sensitive, and must look to software like shared memory rather than a network. The answer is NVLink: a memory-semantic interconnect that today delivers 1.8 TB/s per Blackwell GPU — roughly 14× a PCIe Gen5 link — and rises to 3.6 TB/s per GPU on the Rubin generation shipping into 2026.

The trajectory is a clean doubling per generation, and it is the tightest performance curve in the entire stack. In six years NVLink per-GPU bandwidth has gone up roughly 6×, from 600 GB/s on the A100 to 3.6 TB/s on Vera Rubin. That is not incidental; it is the moat. Competitors can match a GPU’s FLOPS far more easily than they can match a rack that moves a quarter of a petabyte per second internally.

How that bandwidth is delivered is as important as the number. Inside a GB200 NVL72, the 72 GPUs and their NVSwitches are wired through a copper backplane carrying roughly 5,000 cables and over two miles of copper, presenting every GPU to every other GPU in a single hop. NVIDIA chose copper deliberately: doing the same job in optics would have required transceivers and retimers drawing an estimated ~20 kW per rack just to light the NVLink spine. Copper is the moat and the power budget.

The strategic question is whether that moat stays proprietary. Two forces are pushing back. The first is UALink, an open scale-up standard ratified in April 2025 at 200 Gb/s per lane and scaling to 1,024 accelerators per pod, backed by AMD, Intel, Google, AWS, Microsoft, Astera Labs and 85+ members — conspicuously, everyone except NVIDIA. The second is NVIDIA’s own NVLink Fusion, which opens the NVLink fabric to third-party CPUs and custom silicon from Qualcomm, Fujitsu, Marvell and MediaTek. NVIDIA would rather rent its fabric to challengers than let them standardize around an alternative. That is a classic control-point defense.

Adjacent: the memory fabric (CXL)

A fourth fabric is forming alongside compute — for memory. CXL (read “What is CXL?”) lets a rack pool and share DRAM instead of stranding it in each server, attacking the “memory wall” that increasingly bounds inference. CXL 4.0, ratified in November 2025, carries up to 1.5 TB/s on bundled links; memory-pooling deployments target 2026–27, with early benchmarks showing ~20× gains for graph workloads and ~70% for recommendation models. It is the least mature domain — adoption is gated on orchestration and security, not silicon — but if memory disaggregation lands, it becomes a memory-semantic fabric with its own control points. Astera Labs’ Leo controllers are the merchant enabler to watch.

Scale-Out · The Fabric War: InfiniBand’s monopoly is breaking

Once a workload outgrows a single rack, it moves onto the scale-out fabric, and here the story of the last eighteen months is a genuine regime change. As recently as 2023, InfiniBand held roughly 80% of AI back-end networking. It earned that share honestly: InfiniBand is lossless by design, offloads collective operations into the switch via SHARP, and comes pre-integrated with NVIDIA’s stack. For frontier training, it remains the reference platform.

But Ethernet has flipped the market. In 2025, Ethernet more than doubled InfiniBand as the leading fabric for AI scale-out, and by Q4 2025 through Q1 2026 it accounted for roughly two-thirds of switch sales into AI clusters. IDC pegged the datacenter Ethernet switch market at $15.4B in Q1 2026, up 39.8% year-over-year, with NVIDIA — not Arista or Cisco — now ranked #1 by revenue on the strength of Spectrum-X. Amazon, Microsoft, Meta, Oracle and xAI are all building on Ethernet.

This shift has a precedent that should unsettle InfiniBand’s defenders: Ethernet has won this exact fight before, repeatedly. Token Ring, FDDI, ATM, and Fibre Channel each offered a real technical edge behind a proprietary or premium model — and each ceded the volume once Ethernet became merely “good enough” and far cheaper. Open, ubiquitous, and cost-declining beats fast-but-closed on a long enough timeline. InfiniBand is the current chapter, not an exception to the pattern.

InfiniBand’s most durable technical advantage is in-network computing. Collective operations like all-reduce — where every GPU must sum its gradients with every other GPU’s — are the heartbeat of distributed training, and they overwhelm a naively-built network. NVIDIA’s SHARP protocol moves the summation into the switch: instead of shuttling every gradient to every node, the switch aggregates as data flows through it. The result is roughly double the effective all-reduce bandwidth and up to 30% faster job completion on large runs. Ethernet’s answer — distributed, host-based collective libraries plus emerging in-switch offload — is closing the gap but is not yet at parity.

Topology · The Cost of Non-Blocking

Why the wiring diagram is a capital decision

A fabric’s logical shape matters as much as its silicon. Because training traffic is synchronized and all-to-all, any bottleneck anywhere stalls the collective, so large clusters are built as rail-optimized fat-trees: each GPU’s network port is mapped to a dedicated “rail,” so the same-index GPUs that must all-reduce together stay on a single leaf and avoid extra switch hops. This is the reference design at 16,000–100,000+ GPU scale across Meta, Microsoft, and NVIDIA’s own SuperPOD.

The expensive constraint is oversubscription. A training east-west path must be 1:1 non-blocking — any oversubscription stalls NCCL — which means the fabric has to carry every GPU’s full line rate at once. A non-blocking three-tier fat-tree roughly triples switch and optics count versus the oversubscribed networks enterprises tolerate, and that is precisely why networking is 10–15% of cluster cost rather than 2–3%. Topology is not an engineering detail; it is the mechanism that converts the physics of collectives into capital intensity.

The Silicon · Switches, NICs, DPUs

Where bandwidth is concentrated and offloaded

Underneath the protocol wars sits the silicon that actually forwards packets, and it is doubling on a metronome. The switch ASIC is the concentration point of the entire fabric, and Broadcom’s Tomahawk 6, shipping since June 2025, is the first to reach 102.4 Tbps on a single chip — double its predecessor, enough to wire a cluster of more than one million XPUs, with support for both 100G and 200G SerDes and optional co-packaged optics. The cadence is a clean 2× every two years, and it is the reason port speeds march from 800G today to 1.6T by 2027 and 3.2T by 2030.

This is where the merchant-versus-vertical split matters. Broadcom (Tomahawk, Jericho) and Marvell sell switch silicon to everyone; NVIDIA integrates Spectrum-X Ethernet and Quantum-X InfiniBand into its own systems. Broadcom’s combined AI revenue — custom accelerators plus networking silicon — ran $8.4B in Q1 FY26 (+106% YoY), $10.8B in Q2 (+143%), with guidance to ~$16B in Q3. The hyperscalers are deliberately funding a merchant alternative to keep NVIDIA honest.

At the edge of every server, the humble NIC has become a computer in its own right. NVIDIA’s ConnectX-9 SuperNIC delivers 800 Gb/s per port, and each Rubin GPU is paired with 1.6 Tb/s of scale-out bandwidth. But the more consequential shift is the rise of the DPU — a data-processing unit that offloads networking, storage, security and telemetry off the CPU and GPU entirely. NVIDIA’s BlueField-3 already does the work of roughly 300 CPU cores at ≤150W; BlueField-4, launching in 2026, adds 800 TOPS of on-die inference at 800 Gb/s. AMD’s Pensando Pollara 400 AI NIC and Salina DPU give the buyer coalition a credible second source.

Congestion Control · The Invisible Bottleneck

Why rated bandwidth and delivered bandwidth diverge

Here is the layer that separates a fabric’s spec sheet from its real performance. AI traffic is unlike anything Ethernet was designed for: it is synchronized, it moves in enormous “elephant” flows, and in a collective operation thousands of GPUs talk all-to-all at once. Two pathologies follow. Incast — many senders hitting one receiver at the same instant — overflows switch buffers and drops packets. And because classic Ethernet load-balancing hashes each flow onto a single path, a handful of elephant flows routinely collide on the same link, leaving it saturated while parallel links sit idle. In a collective, the slowest path sets the pace for all of them, so tail latency is job latency.

The fixes are the real intellectual property of modern AI networking. Lossless RoCEv2 uses ECN and priority flow control to signal congestion before buffers overflow. But the decisive move is packet spraying — per-packet load balancing that sprays a single flow across every available path — combined with adaptive routing and reorder-tolerant transport. NVIDIA’s Spectrum-X does this in hardware; Alibaba runs Oblivious Packet Spraying; and the Ultra Ethernet Consortium’s UET transport builds packet spraying and credit-based incast control into the standard. Done well, this lifts effective fabric utilization from roughly 50% toward 90%+ — a direct multiplier on the return of every GPU in the building.

Reliability · The Tail That Stalls the Job

At scale, the network’s job is to keep the run alive

There is a failure mode that never appears on a spec sheet yet dominates real economics. At scale, hardware breaks constantly, and synchronous training is brutally intolerant of it. Meta’s public Llama 3 405B run — 16,384 H100s over 54 days — logged 419 unexpected failures, roughly one every three hours, with about half tracing to GPUs, NVLink, or HBM memory. Because the job is synchronous, a single failed GPU or link can halt all 16,384.

Failures scale with component count. Hold the per-GPU rate constant and a 100,000-GPU cluster faces an interruption every ~30 minutes; the million-GPU clusters now being planned face hundreds per day. Meta still achieved >90% effective training time — but only through relentless checkpointing and a fabric that can detect a bad link and reroute around it in hardware. This is why link-level retry, adaptive routing, and fast fault isolation now matter as much as raw bandwidth. Reliability is the fabric’s third axis, alongside bandwidth and latency, and it is the one that gets harder every time a cluster doubles.

Inference · The Workload Inverts

Serving got more network-bound, not less

The single biggest shift in 2026 is that the workload changed. Inference passed half of all AI compute in 2025 and reaches roughly two-thirds in 2026. The lazy inference is that serving needs less network than training. The opposite is happening: modern inference is disaggregated and sparse, and both properties push traffic onto the fabric.

Two forces drive it. First, disaggregated serving splits the prefill and decode phases onto separate GPU pools, which means the multi-gigabyte KV cache must move between them at wire speed — NVIDIA open-sourced NIXL precisely for this, shuttling KV tensors from prefill to decode VRAM over RDMA, RoCE, or NVMe and cutting cost per token 3–5×. Second, mixture-of-experts models — now more than 60% of new open releases — route every token through a sparse all-to-all across experts; DeepSeek’s DeepEP saturates 400G NICs, and a full DeepSeek-R1 forward pass demands on the order of 13.7 TB/s of bandwidth. Inference has become a distributed-systems problem with the network at its center, which is why Dell’Oro projects scale-up Ethernet bandwidth to rise 15× in five years.

The market narrative says inference is cheap and local. The architecture says inference is a bandwidth problem wearing a latency mask.

Optical Networking · The Power Wall

Light is no longer the afterthought — it is the constraint

Every scale-out and scale-across link that exceeds a few meters is optical, which means optics now sit on the critical path of the entire AI build-out — on cost, on supply, and above all on power. A single pluggable transceiver draws about 15W at 800G and ~30W at 1.6T; a large GPU is fed by several of them, and at cluster scale optics consume a double-digit percentage of total AI-factory power. Conventional DSP-based pluggables burn roughly 21 picojoules per bit. When you are provisioning gigawatts, that is real money and real megawatts.

The industry’s response is a march to shorten the electrical path and delete power-hungry silicon. Linear pluggable optics (LPO) remove the DSP entirely, cutting module power 30–50% and shaving latency, and are projected to cover over 40% of short-reach 800G links. Co-packaged optics (CPO) go further, integrating the optical engines directly onto the switch package: NVIDIA claims 3.5× better power efficiency, 4× fewer lasers, and 10× the network resiliency. Broadcom’s Tomahawk 6 “Davisson” is already shipping 102.4 Tbps of optically-enabled switching, and NVIDIA’s Quantum-X Photonics InfiniBand reaches production in early 2026, with Spectrum-X Photonics Ethernet following in the second half.

A different bet: optical circuit switching

Not everyone is buying more packet switches. Google took a structurally different path with optical circuit switching (OCS) — MEMS mirror arrays that steer light from rack to rack without ever converting it to electrons. Its Jupiter fabric has used OCS (codenamed Palomar) across every TPU v4 and v5 pod, and the latest Ironwood generation pairs a 3D-torus with the Apollo OCS. The numbers are hard to argue with: a single OCS draws about 100W versus ~3,000W for an equivalent electrical packet switch — a ~95% reduction — contributing to 40% lower network power, 30% lower cost, and 50× less downtime. An OCS can’t inspect or buffer packets, so it fits the regular, reconfigurable topologies of TPU pods better than general GPU clusters. But it is the clearest existence proof that the packet-switched, convert-at-every-hop model is not the only way to build an AI fabric — and the OCP stood up a standardized OCS project in 2026.

The Money · Market Structure

Sizing the fabric — and who captures it

Pull back to the market map and the numbers are large and fast. Data-center capex is on track to approach $1 trillion in 2026 and hit $1.7 trillion by 2030. Within that, Dell’Oro sees AI back-end switching alone clearing $100B in annual spend by 2030. The optics attach is growing even faster: AI-cluster optical transceivers went from roughly $10B in 2024 to $16.5B in 2025 to a projected $26B in 2026 — ~60% growth two years running — with LightCounting’s upside scenario reaching $100B by 2030. The binding constraint is supply, not demand: InP laser and EML capacity is running roughly 30% short of orders.

The pace is set by a SerDes cadence that doubles lane rate every two-to-three years, dragging port speeds and optics with it. The majority of AI back-end ports have already moved to 800G; Dell’Oro expects 1.6T by 2027 and 3.2T by 2030. Each step is a forced upgrade cycle for switches, NICs, optics, and cabling — a recurring-revenue engine for whoever owns the reference design.

And who is capturing it? NVIDIA’s networking business alone is now running at roughly a $60B annualized rate — Q1 FY27 networking revenue hit $14.8B, up 199% year-over-year, with Spectrum-X past a $10B run-rate on its own. That single line item is larger than Broadcom’s entire AI semiconductor business and dwarfs Arista and Cisco’s AI franchises. The merchant challengers are growing fast — Arista’s AI revenue roughly doubles to $3.25B in 2026, Cisco’s AI infrastructure orders reach $9B — but the scoreboard shows how much of the value the platform owner still captures.

The megacap scoreboard understates where the sharpest returns sit. Beneath NVIDIA and Broadcom is a layer of specialists whose growth rates are, in percentage terms, even steeper — because they sell the connective tissue every architecture needs regardless of who wins the fabric war. Astera Labs (retimers, PCIe/CXL fabric switches, active cables) grew Q1 2026 revenue 93% YoY to $308M, a ~$1.2B run-rate, and now ships a 320-lane scale-up fabric switch. Credo, whose active electrical cables replaced flaky passive copper in hyperscale racks, roughly tripled fiscal-2026 revenue to $1.3B. Marvell’s data-center business — custom XPUs plus Inphi-derived optical DSPs — cleared $6B en route to a targeted $15B by FY28.

The scarcest tier — optical modules and their laser inputs — is where a set of mostly-Asian specialists captures the volume. InnoLight leads 800G at an estimated ~35% share; Eoptolink grew 175% to $1.2B at a 33% net margin; Coherent and Lumentum anchor the lasers and coherent DCI. The top five are roughly half the market. This layer is the cleanest expression of the thesis: pure connectivity exposure and growth that outruns the megacaps — with the offsetting risk that modules commoditize and CPO migrates value back to the switch vendor.

Make the stakes concrete. A fabric is not just a cost line; a better one converts directly into recovered compute. Drag the cluster size below — the bandwidth tax is what lifting delivered throughput from a vanilla ~60% to Spectrum-X’s ~95% is worth every year, alongside the optics bill and the failure interval you are signing up for.

The Investment Lens

Reading the stack as a set of control points

The layers of this stack are not equally defensible, and that distinction is the entire investment question. Sort it on the durable questions — structural advantage, capital intensity, margin durability, replacement risk — and a clear hierarchy of moats appears.

At the top sits the scale-up memory fabric: NVLink is the deepest moat in AI hardware because it is vertically integrated, copper-based, and physically hard to second-source. Just beneath it, and consistently underpriced by the market, is the congestion-control and transport software — adaptive routing, telemetry, and collective offload are what turn rated bandwidth into delivered bandwidth, and they travel with the platform. Switch silicon is durable but merchant, where Broadcom’s scale is the advantage. And optics — the fastest-growing layer — has the least durable margins: modules are commoditizing, the real scarcity is upstream in lasers and EML chips, and CPO is migrating value from the module maker to the switch vendor.

Start with the fabrics themselves. Every interconnect is a different point in a four-dimensional trade space — bandwidth, latency, reliability, and openness — and no fabric maximizes all four. NVLink and InfiniBand win the performance axes; Ethernet wins openness. On a five-year horizon, openness is the axis that has historically compounded.

Put probabilities on it. The one variable that moves the most enterprise value is NVIDIA’s share of AI networking revenue over the next three years — today roughly 65–70%. Base case, weighted at ~55%: Ethernet and open standards keep diffusing, NVIDIA’s platform bundling defends a slipping-but-dominant position, and share settles near half. The bull case (~25%) has NVLink Fusion, Spectrum-X and CPO holding share above 60% because open collectives and transport software lag. The bear case (~20%) has UALink and Ultra Ethernet maturing fast enough that hyperscalers standardize on open scale-up and merchant silicon, dragging share below 45%. Expected value lands around 52–54% — still dominant, but no longer a monopoly.

Bottom Line

The interconnect is the investable layer

The AI narrative has quietly become a networking narrative. Three nested fabrics — scale-up over copper, scale-out over light, scale-across over the metro — now determine whether a warehouse of accelerators behaves like one supercomputer or an expensive pile of idle silicon. The economics have followed: networking is a double-digit share of cluster cost, a ~$60B annual business for NVIDIA alone, and a market Dell’Oro sizes above $100B by 2030.

Own the bottlenecks — scale-up integration, transport software, switch-silicon scale, and upstream optical supply — and underwrite the open-standards coalition as the force that slowly redistributes the rest.

For investors, the signal is to price durability, not growth. The most defensible positions are the vertically-integrated scale-up fabric and the transport software that rides with it; the most durable merchant franchise is switch silicon at scale; and the scarcest physical bottleneck is upstream optical supply — lasers and EML chips — not the modules themselves. The commodity risk sits precisely where the growth headlines are loudest. The single highest-leverage variable over the next three years is whether the open coalition — UALink and Ultra Ethernet — can standardize the scale-up domain the way it already flipped scale-out. That contest, more than any FLOPS number, will decide where the margins in AI infrastructure ultimately settle.

Every AI product you have ever used is, mechanically, the same object: a token generator. Strip away the branding and a chatbot, a coding agent, a search summary, and an image caption are all the identical operation — a trained model run forward to predict the next token, then the next, then the next. That operation is inference, and in 2026 it stopped being a footnote to training and became the entire game. In May 2026 Google disclosed it was processing 3.2 quadrillion tokens per month across its surface area — roughly 38 quadrillion tokens annualized — a 7× increase from 480 trillion a month a year earlier, which was itself up from 9.7 trillion in early 2024. None of that is training. It is the cost of answering.

And yet inference is still treated as a black box: a prompt goes in, an answer comes out, a GPU is invoked somewhere in between. This piece opens the box. We take exactly one prompt and follow it literally — through fifteen physical and logical stops inside a datacenter — and explain precisely what each one does. By the end, “AI infrastructure” should stop being an abstraction and become something you can latency-budget, cost per token, and invest against.

The stakes behind that single round trip are the largest capital reallocation in the history of computing. Inference crossed roughly two-thirds of all AI compute in 2026, up from about one-third in 2023 and one-half in 2025. The four largest hyperscalers guided to ~$725B of 2026 capex, up 77% year over year, more than 60% of it going to power, cooling, and shells rather than chips. The market for inference-specific silicon alone will clear $50B in 2026. The single unit that ties this entire economy together is the token — and the token’s price is falling ~200× per year even as volume grows 7× per year. That scissor is the whole story: demand outruns a cost collapse, which is exactly the condition that turns an input into an economy.

The argument in five points

1. Inference is now roughly two-thirds of AI compute and 80–90% of a model’s lifetime cost. The token, not the model, is the unit of economic output.

2. Two phases, two bottlenecks: prefill is compute-bound (sets time-to-first-token) and decode is memory-bandwidth-bound (sets speed and cost). Nearly every optimization targets one of them.

3. Unit cost is falling ~200×/year while volume grows ~7×/year — Jevons, not deflation. “Enormous” and “high-margin” are different claims.

4. Durable value sits at the physical bottlenecks — memory bandwidth, scale-up interconnect, optics, and power. The inference-software layer is a real but contested business.

5. Buying inference is a weighted trade — latency (TTFT + speed + p99), blended cost, reliability (including the functional-availability gap), security, deployment, model coverage, and portability — scored against your workload.

The machine the economy runs on

The reason to care about the plumbing is economics. Training a model is a capital event — a one-time burn that produces a fixed asset. Inference is the cost of goods sold: it recurs on every query, forever, and it scales linearly with usage. The industry rule of thumb is that inference eventually accounts for 80–90% of the lifetime compute cost of a deployed model. So the shift in Figure 2 — inference passing from a third of compute in 2023 to two-thirds in 2026 — is not a fashion. It is the system reaching steady state, where most of the world’s accelerators spend most of their time serving, not learning.

Two forces make this the defining infrastructure market of the decade. The first is that the frontier moved from bigger models to longer thinking: reasoning and agentic systems spend 10–100× more tokens per query. DeepSeek-R1 matched OpenAI’s o1 while emitting an order of magnitude more tokens per answer; agentic workflows — with their retries, tool calls, and context reloads — run 5–25× more expensive per task than a single-shot completion. The second force is a price collapse without modern precedent: the cost to serve a fixed quality level has fallen at a median ~200× per year since early 2024. Cheaper tokens do not reduce spend; they unlock new workloads faster than price falls. That is the Jevons paradox, and Figure 3 is what it looks like.

So the rest of this article is a single claim, made concrete: every technique below — batching, paging, quantization, speculation, disaggregation, and the network fabric itself — exists to move one number, cost per token at a target latency. Follow the specimen prompt and you will see exactly where each fraction of a cent and each millisecond is won or lost. Fifteen stops. Here is the first.

Step 1: User — the prompt becomes numbers

The journey begins before the datacenter. When you press enter, your client sends text over HTTPS — but the model never sees text. It sees integers. A tokenizer (a byte-level byte-pair encoder) chops the string into sub-word tokens, each mapped to an ID drawn from a vocabulary of roughly 100k–200k entries. Our 12,000-token RFP plus its question becomes a flat sequence of 12,022 integer IDs. Tokenization is deterministic, CPU-side, and effectively free — but it defines the bill. You are charged per token in and per token out, and the model’s context window (200k–1M tokens on 2026 frontier models) is a hard ceiling on how much of that RFP can enter at all.

Step 2: API Gateway — the front door

The request — still just an HTTPS body carrying those IDs — hits the API gateway, the datacenter’s front door. It terminates TLS, parses and validates the request against a schema, enforces API versioning, applies coarse rate limits, attaches a trace ID, and emits the first metering event. It is deliberately dumb and brutally fast: no model logic, just admission and bookkeeping at millions of requests per second, typically on Envoy- or NGINX-class proxies fronted by a web-application firewall. Latency budget: sub-millisecond. Its job is to reject the 5% of traffic that is malformed, over-quota, or hostile before any expensive resource is touched.

Step 3: Authentication — identity, tier, and the meter

Before a single GPU is touched, the request must be attributed. Authentication resolves the API key or OAuth token to an organization, checks its rate-limit tier and spend caps, and decides billing — the per-token price that will apply, whether the caller is eligible for cached-input discounts (a 50–90% reduction we will meet at stop 7), priority lanes, or cheaper batch pricing. It is also the security boundary: org-level data isolation and abuse defense. This stop is where a request stops being anonymous bytes and becomes a metered, priced, and isolated unit of work.

Step 4: Load Balancer — placing the request

Now the work must be placed on hardware. A conventional load balancer spreads requests across many identical model replicas using health checks and live load signals. But for LLM serving, naive round-robin is actively wasteful: two requests that share a long prefix — the same system prompt, the same pasted RFP — should ideally land on the same replica so the expensive prefix computation can be reused from cache. So modern AI load balancing is becoming cache-aware, which is exactly why it blurs into the inference router at the next stop. The load balancer also enforces backpressure: when the fleet is saturated, it is the component that decides whether you queue or get a 429.

Step 5: Inference Router — model, silicon, and kernel selection

The router is where inference gets interesting, and where a startling amount of gross margin is won or lost. In a few milliseconds it answers three questions. Which model? A frontier 70B, a distilled 8B, a reasoning variant, or a speculative draft + target pair — serving a query on a 70B when an 8B meets the service-level objective is pure margin destruction. Which silicon? This is hardware selection, and it is phase-dependent: the compute-bound prefill wants raw FLOPs, while the memory-bound decode wants HBM bandwidth, so the “right” GPU differs within a single request. Which replica? Cache-aware routing sends the request to the instance that already holds this prefix’s KV cache, converting a 12,000-token prefill into a near-free cache hit.

This stop is also home to the compiler / autotuner layer — the systems that, ahead of time, compile and tune the exact GPU kernels for a given model, shape, and chip, and at runtime select the cheapest execution plan. This is not academic: it is a business. Together AI, Fireworks, Baseten, and Modular were each built, in large part, at this layer — turning “which kernel on which chip at which precision” into a product, then monetizing it with their own inference infrastructure. We return to that model, and its valuations, at the end of the journey.

Step 6: Scheduler — the batching engine that funds it all

If a single component makes GPU economics work, it is the scheduler. A lone decode step underuses a GPU catastrophically: at batch size 1, an H100 runs at roughly 30–40% streaming-multiprocessor utilization because it is starved on memory, not math. The scheduler’s answer is continuous (iteration-level) batching: rather than fixed batches, it interleaves many sequences and, at every forward pass, admits freshly-arrived requests and evicts finished ones, keeping the GPU saturated pass after pass. This one trick is the core of vLLM, which delivers 2–4× the throughput of prior serving systems and 3–5× the traffic of a naive PyTorch loop on the same H100.

Step 7: KV Cache — the memory that makes generation possible

Now the request is on a GPU, and we hit the single most important data structure in inference. Inside every attention layer, each token produces a Key and a Value vector. To generate token N+1, the model must attend to the K and V of every prior token — so recomputing them at each step would cost O(n²) and be ruinous at a 12,000-token context. Instead they are cached: the KV cache. Prefill computes and stores K/V for all 12,022 positions once; each subsequent decode step appends one token’s K/V and reads the rest. The KV cache is the largest and most dynamic consumer of GPU memory during serving, and managing it well is most of what a modern inference engine does.

The hard part is fragmentation. Sequences grow to unpredictable lengths, and classic contiguous allocation strands 60–80% of memory. vLLM’s PagedAttention borrows the operating-system trick of virtual memory: the KV cache is split into fixed-size blocks (“pages”), allocated on demand and mapped through a page table, so physical blocks need not be contiguous and finished sequences free their pages instantly. That is the mechanism behind vLLM’s 2–4× throughput — and it is what lets many users’ sequences safely share one GPU.

That sharing is not just a memory optimization — it is a price. If many requests carry the same prefix (a system prompt, a long few-shot preamble, our pasted RFP), its KV cache can be computed once and reused across calls. Providers now sell this directly as prompt / prefix caching: Anthropic charges cached reads at 0.1× input ($0.30 vs $3.00 per million tokens); OpenAI’s GPT-5.x line prices cached input at $0.50 vs $5.00 — both a 90% discount, with roughly 85% lower latency on long prompts. For our specimen, caching the RFP collapses a 12,022-token prefill into a near-free cache hit on every follow-up question.

Step 8: GPU Memory — the real constraint on who you can serve

A serving GPU’s on-package memory holds three things at once: the model weights (fixed), the KV cache (grows with the number of concurrent sequences and their length), and activations and workspace (transient). The weights are the floor; whatever remains is the KV budget, and that budget — not compute — usually caps concurrency. A 70B model is ~140 GB in FP16 and needs two 80 GB H100s just to load; in FP8 it is ~70 GB and fits one GPU with room left for KV cache. This is the first place quantization pays for itself: every gigabyte shaved off the weights becomes KV headroom, and KV headroom is the number of users you can serve simultaneously.

Step 9: HBM — the bandwidth wall and the roofline

This is the counterintuitive heart of the whole machine. Prefill and decode stress entirely different resources. Prefill processes all 12,022 input tokens in parallel — one enormous matrix multiply that saturates the tensor cores. It is compute-bound, and it sets TTFT. Decode is the opposite: to produce each single next token, the GPU must stream the model’s entire weight tensor (plus the growing KV cache) out of HBM, once per token. That read time is fixed by memory bandwidth, not math. At batch size 1, decode’s arithmetic intensity is about 1 FLOP per byte — far below the roofline’s ridge point of ~410–590 FLOP/byte — so the tensor cores sit almost entirely idle, waiting on memory.

So the industry is marching down the precision ladder: FP16 → FP8 → FP4. Fewer bits per weight means fewer bytes to read per token, which directly lifts the memory-bound decode ceiling. NVIDIA’s NVFP4 — a 4-bit floating-point format native to Blackwell’s 5th-generation tensor cores — delivers roughly 2–3× the arithmetic throughput of FP8 and ~1.8× memory reduction while holding within about 1% of baseline accuracy, contributing to end-to-end inference speedups up to 5× versus Hopper. Every bit dropped is bandwidth reclaimed and a token made cheaper — the single most direct lever on cost of goods sold in the building.

Prefill is compute-bound and sets your time-to-first-token; decode is memory-bandwidth-bound and sets your speed and cost. Nearly every technique in inference is an attack on that decode memory wall — batch bigger, quantize smaller, speculate ahead. Read: Why KV Cache and Memory Drive AI Economics.

Step 10: CUDA Kernels — where tokens become arithmetic

The execution plan is now math, and math on a GPU runs as kernels — small programs executed across thousands of cores. The naive way to run a transformer is hundreds of separate kernel launches, each reading its inputs from HBM, computing, and writing results back. For a bandwidth-bound workload that round-tripping is fatal. So the entire craft is kernel fusion: collapse many operations into one kernel that keeps intermediate data in fast on-chip SRAM and touches slow HBM as little as possible. The canonical example is FlashAttention, which computes attention in tiles with an online softmax, cutting HBM reads and writes from quadratic to linear in sequence length for 2–4× speedups. FlashAttention-3 exploits Hopper’s asynchronous engines and FP8 to reach 840 TFLOPS on an H100 — about 85% of peak.

One kernel-level trick deserves its own spotlight because it is the highest-ROI optimization in single-stream serving: speculative decoding. Since decode is memory-bound — the GPU reads all the weights per step no matter what — you can verify several candidate tokens for almost the same cost as generating one. A small, cheap draft model proposes the next K tokens; the large target model checks them all in a single parallel pass and accepts the longest correct prefix. The output is mathematically identical to ordinary decoding, but 2–4× faster. Modern methods like EAGLE-3 push draft acceptance rates above 75%.

Step 11: NVLink — when one GPU is not enough

Our 70B specimen fits on a single GPU, but the frontier does not: trillion-parameter and mixture-of-experts models are sharded across many GPUs, which means the GPUs must talk to each other — on every layer, for every token. This is the point people miss: networking sits directly inside the decode loop, not off to the side. The first fabric is scale-up — NVLink — the ultra-fast interconnect that fuses many GPUs inside a rack into one logical accelerator. NVLink 5 moves 1.8 TB/s per GPU, about 14× a PCIe Gen5 link. The GB200 NVL72 wires 72 Blackwell GPUs and 36 Grace CPUs into a single NVLink domain with 130 TB/s aggregate bandwidth and 13.4 TB of unified memory — one rack behaving as one machine, delivering up to 30× the inference throughput of an H100 cluster on trillion-parameter models, while drawing ~120 kW.

Why is such extreme bandwidth necessary? Because parallelism creates traffic. Tensor parallelism splits each layer’s matmul across GPUs, and after every one the partial results must be summed with an all-reduce collective — many times per token. Worse, mixture-of-experts models route each token to a handful of experts scattered across GPUs, triggering all-to-all communication that frequently becomes the dominant inference bottleneck. DeepSeek’s production stack uses eight 400 Gbps NICs per node and a bespoke library (DeepEP) to overlap this exchange with computation so the GPUs never stall.

Step 12: Switch — the crossbar and the congestion problem

A fabric needs switches. Inside the scale-up domain, NVSwitch is the crossbar: 144 NVLink ports and 14.4 TB/s of non-blocking switching, so all 72 GPUs can talk at full speed at once. Beyond the rack, the scale-out network uses its own switches — NVIDIA’s Quantum-X800 (InfiniBand) and Spectrum-X800 (Ethernet), or Broadcom’s Tomahawk 6 at 102.4 Tbps (64 × 1.6T ports). Two things make an AI switch unlike a cloud switch. First, rail-optimized topology: GPUs are cabled so that the n-th GPU in every server homes to the same “rail” switch, minimizing hops for collectives. Second, in-network computing and congestion control: switches can perform reductions in-fabric (SHARP), and they must absorb the synchronized “incast” of thousands of GPUs finishing a step at the same instant. One congested link stalls an entire collective — the tail-latency problem — so adaptive routing is existential, not optional.

Step 13: NIC — the GPU’s on-ramp to the cluster

A packet leaving the GPU for another rack goes through a network interface card — in 2026, almost always a SmartNIC or DPU (NVIDIA BlueField and peers). Its core job is RDMA over Converged Ethernet (RoCE) or InfiniBand verbs: letting a remote GPU read this GPU’s memory directly, bypassing both CPUs, at 400 Gb/s today and 800 Gb/s becoming standard. But the DPU is a full programmable computer on the NIC: it offloads congestion control, encryption, storage virtualization, and multi-tenant isolation from the host CPU. In a rail-optimized cluster each GPU typically owns a dedicated NIC — DeepSeek pairs eight GPUs with eight 400 Gb/s NICs — so no accelerator shares an on-ramp. This is where “the network” physically begins for a byte, and it is the boundary where our data-gravity theme becomes literal.

Step 14: Ethernet — the scale-out fabric, and the optics beneath it

Historically AI clusters ran on InfiniBand for its ~1–2 µs latency and lossless fabric; Ethernet with RoCEv2 sat at ~5–10 µs and was treated as second-tier for training. That hierarchy is inverting. The Ultra Ethernet Consortium shipped UEC 1.0 in June 2025 — a 560-plus-page rebuild of the Ethernet stack for AI — closing the latency and reliability gap, and Dell’Oro projects Ethernet overtakes InfiniBand in AI back-end networks by 2027. 800G is already the default port speed, with Broadcom’s Tomahawk 6 pushing 102.4 Tbps. For inference specifically — cost-sensitive, multi-tenant, enterprise-adjacent — Ethernet’s economics and open ecosystem are decisive.

Underneath the fabric is its least-discussed cost center: optics. Optical transceivers are roughly 60% of networking cost and 45% of networking power; with networking at ~15–18% of total cluster cost, optics alone are close to 10% of the entire bill. A single 800G pluggable can burn more than 500 W across a switch — more than the switching ASIC itself. The AI optical-transceiver market is running from ~$16.5B in 2025 to ~$26B in 2026 (+57% YoY). The structural fix is co-packaged optics: move the optics onto the switch package, cutting a 1.6T link from ~30 W to ~9 W, with NVIDIA claiming 5× power efficiency and 10× resiliency for its photonics switches shipping in 2H 2026.

For any model too big for one GPU, the network sits inside the decode loop — it runs on every layer, every token. Managing the ~20–40× bandwidth cliff between scale-up (NVLink) and scale-out (Ethernet) is the whole game.

Step 15: Response — detokenize, stream, and settle the bill

The last token is generated, and now the trip runs in reverse. Detokenization maps the integer IDs back into text, and the answer streams token-by-token back out through the NIC, the switch, the load balancer, and the gateway to the user — typically over server-sent events, so tokens appear as they are produced. This is why time-to-first-token dominates perceived speed: the user starts reading at ~0.3 s even though the full answer takes seconds to finish. The final act is the meter closing the bill — input tokens (possibly cached), output tokens (the expensive ones), priced at the tier that authentication assigned back at stop 3. Fifteen stops, one answer.

Final Step: Putting it together: the inference OS

Walk the fifteen stops again and a pattern emerges: the hard problems — batching, KV paging, prefill vs decode, hardware selection, collective communication — are all really one problem, which is scheduling heterogeneous work across a memory-and-network hierarchy to keep expensive silicon saturated. The 2026 answer is disaggregated serving: split the compute-bound prefill and the memory-bound decode onto separate GPU pools, each scaled and tuned independently, and stream the KV cache between them. This is now standard across NVIDIA Dynamo, vLLM, SGLang, llm-d, and Mooncake.

The business of the stack: compilers that became clouds

The most interesting companies on this map sit at stops 5 and 10 — the compiler, kernel, and autotuner layer — and they discovered the same thing: the way to monetize a better kernel is not to license it, but to run it. Own the inference infrastructure, sell tokens, and keep the efficiency delta as gross margin. The logic is airtight. Every 2× you win in kernels, batching, quantization, or speculative decoding is 2× more tokens per GPU — and if you are the one operating the GPUs, that is margin you capture rather than a benchmark you publish. An autotuner that compiles the cheapest kernel for every model × shape × chip is, in effect, a money printer bolted to a fleet.

The growth rates are the argument. Baseten’s run-rate went from ~$200M (Dec 2025) to ~$600M by March 2026 — roughly 1,900% YoY — as it closed a $1.5B round at $11–13B, up from $5B five months earlier. And Modular’s ~$3.9B sale to Qualcomm is the clearest signal of all: a chipmaker buying a hardware-agnostic compiler to attack NVIDIA exactly where the moat is thinnest.

Choosing an inference provider: the real decision criteria

If you are buying inference rather than building it, the fifteen-stop machine collapses into a purchasing decision — and the most common mistake is optimizing for a headline price per million tokens. There is no best provider in the abstract; there is only the best provider for a workload shape. A real-time voice agent, a nightly batch summarizer, a multi-step autonomous agent, and a HIPAA-bound enterprise app weight the same criteria completely differently. The framework below is the one that survives contact with production.

The three everyone names — measured correctly

Latency is two numbers, not one. Time-to-first-token (the prefill latency) governs how fast the answer starts; inter-token latency, or output speed, governs how fast it finishes. A provider can win one and lose the other, and the number that matters for user experience is the p99 tail, not the median — a great p50 with a poor p99 still produces visible stalls. Typical GPU inference lands first tokens in ~400–600 ms; the speed-silicon clouds compress that sharply, with Groq and Cerebras posting sub-100–150 ms TTFT and output speeds of 1,600–2,100+ tokens/second on Llama-70B-class models — roughly 4–6× a typical GPU stack.

Cost is blended, not posted. The list price is two numbers — input and output per million tokens — and output is priced ~4–5× input and usually dominates. Your real unit cost depends on your input:output ratio, your cache-hit rate (prompt caching cuts input 50–90%), and whether the work can use a batch tier (often ~50% cheaper). Posted rates in mid-2026 span two orders of magnitude — from roughly $0.04–$0.20 per million on cost-optimized open-model endpoints (DeepInfra; Groq at $0.15/$0.60) to several dollars per million for frontier models — so two providers with identical headline prices can differ 3× on your traffic.

Reliability is more than an uptime percentage. The formal SLA (Azure OpenAI publishes 99.9% for token generation; enterprises increasingly demand a latency SLA such as sub-200 ms TTFT for 99.99% of calls) is necessary but not sufficient. The failure modes that actually break products sit outside it: refusal spikes, silent model-version bumps that regress your evals, and quota-driven throttling under load. A vendor can honor 99.95% uptime and still degrade your application through any of these — which is why the mature answer is to pin model versions, negotiate capacity, and monitor functional availability yourself.

The seven everyone forgets

Beyond latency, cost, and reliability, seven criteria decide production fit. Throughput & rate limits (tokens-per-minute ceilings and burst headroom) gate agentic fan-out and scale. Security & compliance — SOC 2 Type II, HIPAA, ISO 27001, GDPR — is table stakes in regulated industries and slow to retrofit. Data residency & private deployment (zero-retention guarantees, VPC/BYOC, on-prem) is what clears enterprise procurement; most consumer-grade endpoints leave retention vague and offer no private path, quietly disqualifying themselves. Determinism & version control — fixed seeds, pinned checkpoints — protects your evaluation suite from silent drift. Model coverage & freshness (breadth, day-zero support for new open weights, fine-tune/LoRA hosting) sets how fast you adopt the frontier. Deployment flexibility (serverless vs dedicated vs self-host) sets your cost/control trade. And portability — an OpenAI-compatible API and clean multi-provider routing — is your insurance against every one of the above going wrong.

Pick your workload column first, then shop. A headline price per million tokens is close to meaningless without your input:output ratio, your cache-hit rate, and a latency SLA that covers the p99 tail.

Implications: where the value accruescore

Three conclusions follow. First, the physical bottlenecks compound: HBM bandwidth sets the decode ceiling, the NVLink scale-up domain is genuinely proprietary, and optics plus power are the scarce inputs — more than 60% of that $725B capex is already power and buildings, which makes tokens per watt the terminal metric. Second, the network is bifurcating: scale-up stays closed and defensible while scale-out goes open (Ethernet/UEC) and commoditizes, so the edge is the scale-up domain, optics/CPO, and congestion IP — not commodity switching. Third, the inference-software layer is real but contested: margin equals efficiency delta × utilization × scale, durable only where a player has converted speed into distribution and switching costs before the price collapse makes raw performance table stakes.

The decisive read: through 2027, back the bottlenecks — memory and bandwidth, scale-up interconnect, optics and power — and the inference platforms that have already turned efficiency into lock-in. Be skeptical of anything whose entire pitch is raw speed, because raw speed is exactly what the compiler layer, and NVIDIA’s own free tooling, are busy commoditizing. The token economy will be enormous — from 9.7 trillion to 3.2 quadrillion Google tokens a month in two years is not a curve that reverses — but enormous and high-margin are different claims, and the fifteen stops are where that difference is decided.

Bottom line: Back the physical bottlenecks — memory bandwidth, scale-up interconnect, optics, power — and the platforms that have turned efficiency into lock-in. Be skeptical of any pitch whose only moat is being faster; that is exactly what the compiler layer and NVIDIA's own free tooling commoditize.

Follow one token and the whole economy becomes legible.

Sources & further reading — accessed June 2026

1. Deloitte — More compute for AI, not less (inference ~2/3 of compute, 2026)

2. MarketsandMarkets — AI Inference Market size & forecast ($106B→$255B)

3. Tom’s Hardware — Big Tech AI capex reaches $725B in 2026

4. Introl — Hyperscaler capex & AI infrastructure

5. Shacknews — Google processing 3.2 quadrillion tokens/month

6. Crypto Briefing — Google 3.2Q tokens/month, 7× YoY

7. Google — Sundar Pichai, I/O 2026 keynote

8. Epoch AI — LLM inference price trends

9. TokenCost — AI Price Index (300× drop, 2023–26)

10. NVIDIA / SEC — Q1 FY2026 results ($39.1B DC)

11. Intellectia — NVIDIA record $81.6B quarter (May 2026)

12. Kwon et al. — PagedAttention / vLLM (arXiv)

13. Spheron — Continuous batching, PagedAttention, chunked prefill

14. RunPod — vLLM PagedAttention & continuous batching

15. Hao AI Lab (UCSD) — DistServe: prefill–decode disaggregation

16. Hao AI Lab — Disaggregated inference, 18 months later

17. NVIDIA — Introducing Dynamo (30× on DeepSeek-R1)

18. NVIDIA — Inference cluster economics & AI-factory tokens/s

19. Local AI Master — Speculative decoding: EAGLE, Medusa

20. Spheron — EAGLE-3 speculative decoding (2–4×)

21. NVIDIA — Introducing NVFP4 low-precision inference

22. Edge AI + Vision — Blackwell: impact of NVFP4

23. EmergentMind — Triton & CUDA kernels; FlashAttention-3

24. Spheron — HBM3e/HBM4 bandwidth & roofline for inference

25. Samanvya Tripathi — Prompt caching: 90% cost cut

26. PromptHub — Prompt caching across OpenAI/Anthropic/Google

27. NVIDIA — NVLink & NVSwitch (1.8 TB/s, 14.4 TB/s)

28. NVIDIA — GB200 NVL72 (72 GPUs, 130 TB/s)

29. Introl — NVLink & scale-up networking

30. Ultra Ethernet Consortium — UEC 1.0 launch

31. HPCwire — Ultra Ethernet has arrived

32. TrendForce — InfiniBand vs Ethernet scale-out

33. SemiAnalysis — Co-packaged optics: scaling with light

34. NVIDIA — Silicon photonics for agentic AI

35. TrendForce — AI optical transceiver market to $26B (2026)

36. DeepSeek — DeepEP expert-parallel comms library

37. Introl — Mixture-of-experts infrastructure

38. Introl — Inference-time scaling & reasoning models

39. Sacra — Fireworks AI revenue & valuation

40. SiliconANGLE — Fireworks $250M Series C at $4B

41. Startup Fortune — Baseten $1.5B round at $11–13B

42. Contrary Research — Baseten company profile

43. Sacra — Modular valuation & funding

44. Startup Fortune — Qualcomm to buy Modular (~$3.9B)

Underneath the anecdotes sit the numbers. Claude Code now authors more than 2.6 million commits a week — more than every other AI coding tool on earth combined — and that counts only public GitHub; the enterprise majority of its work runs in private repositories the tally never sees. Inside Anthropic, Claude already writes north of 80% of the code that ships to production, up from low single digits when the tool launched. Nine months after release it became the fastest software product in history to a $1 billion run-rate — and it is now several multiples beyond that, still accelerating.

The lazy reading is “the models got good.” The useful reading is structural. Of everything we ask machines to do, coding fell first — and the reason has little to do with how hard programming is and everything to do with grading. Software is the one discipline that ships with its own referee: a compiler that rules valid or invalid, a test suite that rules correct or incorrect, instantly and for free. That objective signal is the single scarcest ingredient in modern AI training — and almost no other kind of work supplies it. Claude Code is what happens when a frontier lab aims reinforcement learning at the one domain where the answer key already exists, then builds a harness good enough to act on a real computer. The rest of the economy is now a question of who can write the next answer key.

Why code was the first domino

Modern language models are pre-trained on text, then post-trained to be useful. The decisive lever in post-training is reinforcement learning: the model proposes, an environment scores the proposal, and the scores reshape the weights. The bottleneck is the scorer. For most knowledge work — write a strategy memo, summarize a deposition, design a campaign — there is no cheap, objective function that returns right or wrong. You need humans, and humans are slow, expensive, and inconsistent at scale.

A single Claude Code turn: It reads the repo, edits the source, runs the test suite, and only ships when the suite goes green — the same read → act → execute → verify loop it was trained on:

Code is the exception. Correctness is executable. You can run the program, run its tests, and read an unambiguous pass or fail — millions of times, in parallel, for the marginal cost of compute. This is the substrate behind the term of art reinforcement learning with verifiable rewards (RLVR): rather than training a model to match human preference, you train it against ground truth a machine can check. Code, math, and formal proofs are the domains where that ground truth is effectively free. Software has the largest and richest supply of it — every repository on earth is a corpus of problems with graders already attached.

How the loop is actually built

The mechanics are a loop, not a single forward pass. Start from a real task — a GitHub issue against a real repository at a known commit. The policy (the model) is dropped into a sandbox and samples a rollout: it reads files, greps the codebase, edits source, and runs commands across many turns. When it judges itself done, the harness runs the repository’s test suite. Tests pass, reward is 1; tests fail, reward is 0. That scalar — or a denser version that credits compilation, partial progress, and fault localization — is fed back through a policy-gradient algorithm (PPO- or GRPO-style) that shifts the weights toward the behaviors that earned reward. Repeat across millions of tasks and the model stops emitting plausible-looking code and starts producing code that runs.

Two properties make this potent. First, no human sits in the inner loop, so the data engine scales with compute rather than headcount — the defining economic feature of the approach. Second, the reward is grounded in execution, not in what looks correct, which is why benchmarks like SWE-bench Verified — drawn from real, resolved GitHub issues — became the industry’s yardstick. The failure mode is reward hacking: a model that learns to overwrite the test instead of fixing the bug. Anthropic has said it cut reward hacking in the Claude 4 generation specifically by improving environments, sharpening reward signals, and monitoring with prompted classifiers and execution checks that flag tampered tests. Clean reward design, not raw scale, is the hard part.

The harness is the product

A model that passes tests in a sandbox is necessary but not sufficient; it still has to operate a real computer. Anthropic’s second insight was that the interface between agent and machine — what they call the Agent-Computer Interface (ACI), deliberately analogous to the human-computer interface — is a first-class engineering surface. In building their SWE-bench agent, the team reported spending more time optimizing tools than the prompt. Rewriting tool descriptions so the model could call them reliably cut task-completion time by roughly 40%.

The philosophy is primitives over integrations. Claude Code ships a deliberately small toolset — read a file, list a directory, grep, make surgical edits, run a Bash shell — and lets the model compose them, rather than wrapping every external system in a bespoke connector. The same instinct produced the Model Context Protocol (MCP), Anthropic’s late-2024 open standard for plugging tools and data into agents, now broadly adopted across the industry. The strategic point compounds: the model is the engine, but the harness is what converts raw capability into reliable work — and it is far harder to clone than a benchmark score.

That moat extends one layer upstream, into the environments themselves. The harder it becomes to verify long-horizon work, the more the binding constraint shifts from the model to whoever can build and grade the training environments — the argument I make in RL Environments for Agentic AI, where verification, not raw model capability, is the real prize. Anthropic is already the single largest buyer of coding and computer-use environments — on the order of tens of millions of dollars a year, and growing an estimated 3–5x into 2026. It is not only training on verifiable rewards; it is buying up the supply of verification itself.

The model is the engine. The harness is the moat. One converges across labs; the other is years of accumulated environment and tool design.

Crossing the human-competitive threshold

The flywheel shows up directly in the scores. On SWE-bench Verified, Claude moved from 49% with 3.5 Sonnet in late 2024 to 80.9% with Opus 4.5 in November 2025 — the first model past 80%, a level above the human-competitive bar on the benchmark — and to 88.6% with Opus 4.8. The gains arrived with falling cost: Opus 4.5 matched the prior Sonnet’s best result using roughly three-quarters fewer output tokens. Just as important is task horizon. METR’s tracking shows the length of tasks agents can complete autonomously doubling every four to seven months, and accelerating. Each doubling moves the frontier from autocompleting a snippet, to refactoring a module, to owning a multi-day migration — the exact shift that turns a coding assistant into a coding agent.

The second-order effects are louder than the benchmark. Inside Anthropic, the share of production code written by Claude went from low single digits at launch to more than 80% by May 2026; the typical engineer now merges roughly 8x more code per day than in 2024, and on the hardest, spec-light problems — where no clean test exists to begin with — Claude’s success rate jumped about 50 points in six months, to 76%. In one April 2026 stretch, Claude shipped 800-plus fixes that cut a single class of API errors by a factor of 1,000. The recursive part is the part to sit with: Claude Code now writes most of the code for the next version of Claude. The loop that cracked coding is increasingly pointed at itself.

The fastest revenue curve software has produced

Capability converted into the steepest revenue curve software has produced. Claude Code reached a $1 billion run-rate six months after its May 2025 launch — faster than any commercial software product before it, versus roughly eleven months for ChatGPT and years for the canonical SaaS breakouts — then doubled to $2.5 billion three months later. It is the engine under Anthropic’s own ascent: company run-rate went from ~$87 million entering 2024 to $1 billion by year-end, $9 billion exiting 2025, and run-rate tracking past $55 billion by late June 2026 (above the $50B-plus Anthropic guided). The valuation kept pace — a $380 billion Series G in February became a confidential IPO filing at ~$965 billion on June 1, after a $65 billion Series H, with the company on pace for its first profitable quarter. Coding is the throughline: Anthropic holds an estimated 54% of coding-specific LLM spend, enterprise is now more than half of Claude Code revenue with business seats up roughly 4x year-to-date, and the base is contracted at the top — 8 of the Fortune 10 and ~70% of the Fortune 100 are customers, with Accenture alone training 30,000 staff on Claude.

What Claude Code is actually running at now

Anthropic’s last public figure — $2.5B, in February — is four months stale in the fastest-moving product in software. Three independent methods say the truth is multiples higher.

By cadence: Claude Code went $1B → $2.5B in the three months to February — 2.5x in a quarter. Even decelerating hard, to ~25% a month, that lands near $7–8B by late June. By revenue share: Claude Code was ~18% of company run-rate in February ($2.5B of $14B); with company run-rate tracking past $55B by late June and coding the lead driver, a steady 15–18% share is $8–10B. By third-party forecast: FutureSearch models Claude Code near $21B by May 2027 — a path that runs through roughly $8–10B today. The three converge on a best estimate of ~$8–9 billion in annualized run-rate as of mid-2026 — about 3.4x the last disclosed number in four months — with a bull case above $12B and a year-end exit in the $15–20B range. Even the floor of that band makes Claude Code, by itself, one of the five largest developer-software businesses on earth, built in thirteen months. The $2.5B headline you still see quoted understates the product by more than half.

Methodology — how we get to ~$8.5B
Two disclosed anchors ($1B Nov '25, $2.5B Feb '26), then three methods — cadence (~$8B), revenue share (~$8–10B), independent forecast (~$8–10B) — converging on ~$8–9B.

Past the entire pre-AI developer-tools market

Set Claude Code against the franchises that defined developer tooling before AI. GitHub, after sixteen years and a Microsoft acquisition, runs about $2 billion a year. GitLab does roughly $580 million; JetBrains, around $250 million. The entire standalone developer-tools software market was worth near $5–6 billion in 2024. Claude Code — which did not exist commercially before mid-2025 — was already past $2.5 billion at its last disclosure and, on the estimate above, is running at ~$8–9 billion today: more than GitHub, GitLab, and JetBrains combined, and larger than the whole pre-AI category by itself. The AI-native peers are filling the same vacuum — Cursor crossed $2 billion ARR in the same window — but the structural story is the repricing, not the ranking. Seat-based licensing, the model that underwrote two decades of developer software, is giving way to consumption metered against tokens, and value is migrating from the editor a human drives to the agent that does the work.

Why the margin moves to whoever owns the loop

The repricing has a unit-economic engine underneath it. An agent-assisted task costs roughly $6–7 of tokens against a fully-loaded knowledge-worker day of $350–500 — a 50–80x cost ratio before counting any quality or speed gain. And the cost side is deflating fast: Opus 4.5 hit the prior generation’s best SWE-bench score using about 76% fewer output tokens, so the cost to resolve a given issue is falling several-fold a year even as capability climbs. That is the inverse of the SaaS cost curve, where headcount and gross margin are roughly fixed. The ~75% gross margin seat-based software enjoyed was, in effect, a toll on human workflow; agents route around the workflow, and the surplus accrues to whoever owns the loop and the tokens beneath it.

Coding is the beachhead, not the destination

This generalizes beyond software because the loop does. Strip Claude Code to its pattern — read unstructured context, apply domain knowledge, produce structured output, verify against a standard — and you have the shape of most information work, on the order of a billion information workers and a $15-trillion-plus economy. Anthropic has already aimed the same architecture at general computing with Cowork, built by four engineers in ten days, most of its code written by Claude Code itself. The binding constraint is not model IQ; it is whether a task admits a cheap, trustworthy grader. Where one exists or can be built, expect the coding playbook to repeat; where it doesn’t, human judgment stays in the loop longer.

The bear case is specific, not vague. Verification debt: verifiable rewards optimize for passing the grader, not for what humans actually wanted, and as task horizons lengthen the graders get harder to write — reward hacking is reduced, not solved. Benchmark mirage: SWE-bench scores are compressing at the top (80.9% to 88.6% in months) and partly reflect contamination and scaffold tuning, so real-world generalization may trail the leaderboard. Capital intensity: the moat is expensive to hold — growth is now gated by compute and power, gross margin rides inference-cost and cloud-pricing curves Anthropic only partly controls, and the same RLVR recipe is available to every well-funded rival, so model leads are measured in months. The durable edge, if there is one, is not the model; it is the compounding system around it — proprietary verifiable environments, a hardened harness, enterprise distribution, and consumption pricing that turns usage into revenue.

Whoever writes the grader wins the category

The lesson of Claude Code is narrower and more useful than “AI is good at coding.” Software fell first because its grader was already written — the compiler and the test suite predate the model by decades. The verifiable-reward loop turned that free signal into capability, the harness turned capability into reliable work, and consumption pricing turned reliable work into the fastest revenue ramp software has seen. The next domains to fall will be the ones where someone writes the grader. The labs that build those environments — not merely the ones that ship bigger models — will capture the outcome.

Sources & notes

1. SemiAnalysis, “Claude Code is the Inflection Point” (Feb 2026) — 4% of public commits; ~135k commits/day; 20% projection; ACI and harness discussion. link

2. VentureBeat — Anthropic hits $30B run-rate after “crazy” 80x growth. link

3. Simon Willison — Anthropic run-rate revenue reaches ~$47B (May 2026). link

4. Anthropic — “Anthropic acquires Bun as Claude Code reaches $1B milestone” (fastest product in software history to $1B run-rate, ~6 months). link

5. Anthropic — Series G: $30B raised at $380B post-money valuation. link

6. CNBC — Anthropic tops OpenAI as most valuable AI startup (May 2026). link

7. MindStudio — Claude Code at ~$2.5B annualized (doubling in three months). link

8. Vellum / Abaka AI / Vals.ai — SWE-bench Verified: Opus 4.5 at 80.9% (first >80%), Opus 4.8 at 88.6%; competitor comparison. link · link

9. Anthropic — “Claude 3.7 Sonnet and Claude Code”: SWE-bench Verified 62.3% standard / 70.3% with scaffold. link

10. Anthropic engineering — SWE-bench agent: tool optimization, primitives over integrations, ~40% task-time reduction from tool-description rewrites; Model Context Protocol.

11. Related reading (author) — Zeoli, C., RL Environments for Agentic AI: Who Will Win the Training & Verification Layer by 2030, Data Gravity — source for the verification-as-bottleneck thesis and Anthropic’s RL-environment spend (tens of millions/yr, ~3–5x growth into 2026). link

12. METR — autonomous task horizons doubling every ~4–7 months (cited via SemiAnalysis).

13. getlatka / brandhistories — GitHub ~$2B run-rate; JetBrains ~$250M. link · link

14. GitLab Form 8-K (FY2024) — total revenue ~$579.9M. link

15. Software development tools market size (~$5–6B, 2024) — Verified Market Research; Cognitive Market Research. link

16. JetBrains April 2026 developer survey — Claude Code “most loved” (46%); at-work usage 3%→18%. link

17. VentureBeat — Anthropic: 80%+ of new production code authored by Claude; engineers merge ~8x more code/day; 76% success on hardest spec-light problems (+50 pts in six months). link

18. Anthropic — “When AI builds itself” (recursive self-improvement; share of merged code; 800+ fixes cutting one API-error class ~1,000x). link

19. Commit-share data (0.7% Sep ‘25 → 4.5% Mar ‘26; ~2.6M commits/week; 19M+ public commits; more than all other AI coding tools combined) — SemiAnalysis; serpsculpt; “Claude’s Code” tracker (Hacker News). link · link

20. Adoption / enterprise datapoints (Fortune 100, Accenture 30k, 54% coding LLM spend, enterprise >50% of Claude Code revenue, business seats ~4x YTD) — Menlo; Accenture newsroom; getpanto; TechnologyChecker. link · link

21. Fortune — Anthropic confidentially files for IPO at ~$965B after a $65B Series H (June 1, 2026); on pace for first profitable quarter. link

22. TechCrunch — Anthropic in talks to raise ~$50B at ~$900B (April 2026). link

23. FutureSearch — Anthropic financial forecast; Claude Code path toward ~$21B by May 2027 (input to the run-rate estimate in Fig. 6). link

24. Unit economics — agent-task (~$6–7) vs. loaded knowledge-worker day (~$350–500) and Opus 4.5 token efficiency (~76% fewer output tokens): SemiAnalysis; Anthropic. link

25. Figure 6 is the author’s estimate, not an Anthropic disclosure — triangulated from growth cadence, revenue-share, and the FutureSearch path. Treat the $6–12B band as the honest uncertainty.

The shift is measurable. On OpenRouter — the largest neutral router of LLM traffic, processing north of 20 trillion tokens per week — Chinese-origin models moved from a rounding error to the majority of consumption inside eighteen months. Anthropic and Google still command premium revenue per token, but on raw usage the center of gravity has moved offshore. This is what commoditization of a layer looks like in real time: the frontier becomes a feature, and the open, cheap, good-enough tier eats the volume.

This piece maps the field in three parts, plus the reframe the whole story turns on: (1) the labs and their relative share of use — Qwen, DeepSeek, Kimi, and the Alibaba/ByteDance platforms beneath them, including a deeper profile of who they are and what they are worth; (2) the chip ecosystem and supply — Huawei Ascend, the HBM bottleneck, and the offshore-neocloud workaround now being closed; and (3) ByteDance’s lead in video and media models, the one frontier where a Chinese lab is not catching up but setting the pace. Threaded through all three: the January 2025 DeepSeek panic predicted the wrong disruption.

Key points

• Distribution flipped. By May 2026, Chinese open-weight models were ~61% of all tokens consumed on OpenRouter, the largest neutral LLM router. Four of the five most-used models are Chinese — and Meta’s Llama, the open-weight leader two years ago, has fallen off the rankings entirely.

• The DeepSeek panic was backwards. Markets feared a cheap Chinese model would crater compute demand — they wiped ~$600B off Nvidia in a day. Nvidia is now worth $5.14T (+50% YoY). Semis didn’t break; the disruption landed on model market share instead.

• Price is the weapon. DeepSeek-V4-Pro lists ~12× under GPT-5.5 at comparable benchmark intelligence. Cost, not capability, is the wedge.

• The labs are a portfolio, not a monolith. Qwen owns the ecosystem, DeepSeek price-performance, Kimi agentic cadence, ByteDance media. The pure-play labs alone are now worth ~$160B combined; their platform backers add >$1.5T.

• Supply is HBM-bound, not logic-bound. SMIC can feed Huawei’s Ascend line; domestic HBM caps 2026 output near 250k–300k 910C-class packages. The binding constraint is memory.

• The offshore-compute loophole closed. On May 31, 2026, BIS extended licensing to any China-parented buyer regardless of where its subsidiary sits — shutting the Singapore/Malaysia neocloud channel for new purchases.

The share shift: China captured the open layer in eighteen months

The single most important chart in Chinese AI is not a benchmark — it is a usage curve. OpenRouter’s token-routing data shows Chinese open-weight models crossing from negligible to majority share of all tokens processed between late 2024 and mid-2026. The inflection traces directly to DeepSeek-R1 in January 2025, which proved that a Chinese lab could match Western reasoning quality and release the weights, and to the subsequent flood of Qwen, Kimi, GLM, MiniMax, and Xiaomi MiMo releases that filled every price-performance niche beneath the U.S. frontier.

Two structural forces sit under the curve. First, workload mix shifted toward code: programming rose from ~11% of OpenRouter usage at the start of 2025 to more than 50% by mid-2026, and Chinese models are disproportionately strong and cheap on coding. Second, the router itself scaled ~4× year-over-year — from roughly 5T tokens/week in April 2025 to over 20T in April 2026 — so China did not just take share, it took share of a market that quadrupled.

Aggregate share understates the drama at the top of the table. The incumbents did not merely lose ground proportionally; Google’s token share collapsed from roughly 37% to 13% over the year as developers re-routed cost-sensitive workloads to open weights. Anthropic held the premium tier on reasoning and coding quality, and OpenAI’s share of routed volume — distinct from its first-party ChatGPT traffic — fell into single digits.

Read the vendor table closely and three things stand out. First, the single largest vendor is Xiaomi — not a name most Western investors associate with frontier AI — whose MiMo models alone take ~21% of routed tokens and ~22% of all coding traffic. Second, DeepSeek at ~17.6% outranks every U.S. lab, sitting above Anthropic’s 15.4% and well above Google and OpenAI. Third, Meta has effectively exited: the company that defined open-weight AI with Llama in 2023–24 has fallen below 1% of routed volume, displaced entirely by Chinese open weights. (Methodologies differ — counting only identified vendors puts the Chinese share near 56%, while the broader origin-based measure in Exhibit 1 reaches ~61% — but every cut tells the same story.)

The frontier is no longer where the volume is. Premium reasoning is a 15-point niche; the cheap, open, good-enough tier is the market — and it is overwhelmingly Chinese.

One caveat worth stating plainly: routed inference is not the whole market. First-party consumer traffic — ChatGPT, Gemini, Doubao — dwarfs the router, and Anthropic and OpenAI still capture the majority of industry revenue because they price at a premium and serve enterprises directly. OpenRouter is a leading indicator of where developers reach when nobody is watching the brand. On that signal, the open Chinese tier is the default.

The DeepSeek panic predicted the wrong disruption

It is worth remembering what the market actually believed eighteen months ago. On January 27, 2025, days after DeepSeek-R1 showed a Chinese lab could match Western reasoning for a fraction of the training cost, investors concluded the obvious: if frontier intelligence is suddenly cheap, the demand for expensive accelerators must collapse. They sold. Nvidia fell 17% and shed roughly $600 billion in market value in a single session — the largest one-day loss in U.S. market history — and Broadcom dropped 17% (~$200B). The thesis was crisp: efficient Chinese models break the compute supercycle.

That thesis was wrong in the most complete way possible. By June 2026, Nvidia is worth $5.14 trillion, up ~50% year-over-year, the most valuable company on earth, and AI capex hit records on both sides of the Pacific. Cheaper intelligence did not shrink compute demand — it expanded it. Token volume on OpenRouter alone quadrupled; total inference grew faster than per-token efficiency improved. The Jevons dynamic held: drop the unit cost of intelligence and the world consumes vastly more of it.

What actually came true was the other half of the DeepSeek thesis — the half the market ignored while it was selling chips. DeepSeek did not destroy demand for compute. It demonstrated that Chinese labs would capture the model layer, and they did: from ~2% to ~61% of routed tokens. The disruption was real, but it landed one layer up from where the market priced it — on model-vendor share, where margins were already thin and are now mostly Chinese, not on silicon, which compounded. For an infrastructure investor, that is the single most important re-rating of the cycle: efficiency gains accrue to the compute layer and the distribution layer; they get competed away in the middle.

The Chinese Labs: A portfolio of specialists, not a single champion

Treating “Chinese AI” as one actor obscures the real structure. It is a portfolio of at least seven serious franchises, each with a different control point, business model, and owner. Three pure-play open-weight labs — DeepSeek, Moonshot (Kimi), and Zhipu (GLM) — compete on weights, price, and release cadence. Two platform giants — Alibaba (Qwen) and ByteDance (Doubao/Seed) — treat models as demand generation for cloud and consumer surfaces. And two more — MiniMax and Xiaomi (MiMo) — have become major suppliers almost sideways, the first out of a content-generation business, the second out of a phone company. What they share is a strategy: release open weights, price at a fraction of the U.S. frontier, and let distribution compound.

The aggregate is the statistic that should reframe the category for any investor still treating these as research curiosities. The four pure-play Chinese AI labs — Zhipu, DeepSeek, MiniMax, and Moonshot — are now worth roughly $159B combined, up from a near-zero base eighteen months ago. DeepSeek alone went from a quant fund’s side project to a reported ~$50B first external round (founder Liang Wenfeng personally subscribing ~$3B, ~40% of it). Behind them stand platform owners — Tencent, ByteDance, Alibaba — carrying more than $1.5 trillion of AI-exposed market value.

Qwen / Alibaba — the ecosystem standard

Qwen is the most important open-model franchise in the world, and the gap is not close. It has surpassed 1 billion cumulative downloads on Hugging Face, overtaking Meta’s Llama as the most-downloaded open model, and now anchors more than 200,000 Qwen-tagged models and over 113,000 derivatives — more than Google and Meta’s base families combined. Roughly 40% of all new LLM derivatives created on Hugging Face are now Qwen-based, and China accounted for ~41% of all Hub downloads over the trailing year. In June 2026 trending rankings, Chinese open models held five of the top ten slots.

The strategic logic is not philanthropy. Alibaba runs Qwen as a top-of-funnel for Alibaba Cloud: open weights seed adoption, paid inference and fine-tuning capture the monetizable tail. It is working. Cloud Intelligence Group revenue reached ¥158.1B in FY2026 (+34% YoY), AI-related product revenue posted triple-digit growth for eleven consecutive quarters, AI now drives ~30% of external cloud revenue (a ~$5B run-rate), and the firm pushed capex to ¥126.1B for the year. Open weights are the cheapest customer-acquisition channel in infrastructure.

DeepSeek — the price-performance frontier

DeepSeek is the lab that started the repricing. Founded in July 2023 by Liang Wenfeng as an offshoot of his quantitative hedge fund High-Flyer, it has no consumer platform to defend and no cloud P&L to protect — just a model franchise, a stockpile of GPUs the fund had accumulated for trading, and a founder who treats open weights as ideology. Its R1 release in January 2025, trained for a reported <$6M on export-compliant H800s, is what triggered the Nvidia selloff. In 2026 it took its first outside capital at a reported ~$45–50B valuation, with Liang personally subscribing ~$3B — a structure that keeps control firmly with the founder.

The product strategy is to undercut. The V4 family (April 2026) spans V4-Pro (1.6T parameters, 49B activated) and V4-Flash (284B / 13B activated), both at 1M-token context. List pricing of $0.435 / $0.87 per 1M input/output tokens places it roughly 3× below Gemini 3.1 Pro and ~12× below GPT-5.5 at comparable benchmark intelligence. The result is measurable U.S.

pull: DeepSeek’s share of token usage on Vercel rose from under 1% to ~17% by May 2026, even as its share of revenue stayed near 1% — the clearest single illustration of how cheap the open tier has made intelligence.

Kimi / Moonshot — agentic coding and cadence

Moonshot, founded in March 2023 by Tsinghua- and Carnegie Mellon-trained researcher Yang Zhilin, competes on two axes incumbents struggle to match: coding quality and shipping speed. Kimi K2.6 (April 20, 2026), a 1T-parameter MoE, ties GPT-5.5 on SWE-Bench Pro (58.6%) and leads Humanity’s Last Exam with tools (54.0) — ahead of GPT-5.4, Claude Opus 4.6, and Gemini 3.1 Pro on that benchmark — at roughly 80% lower cost, with an agent-swarm architecture that coordinates up to 100 parallel sub-agents. Artificial Analysis ranks it #1 among open-weight systems. Then came K2.7-Code on June 12 — the fifth major Kimi release in under a year, cutting reasoning-token usage ~30%. That cadence is the moat: Moonshot iterates faster than enterprise procurement cycles.

The market is paying for it. Moonshot raised $2B at a $20B valuation in May 2026 (cumulative ~$3.9B in six months, the most of any Chinese LLM startup), with Alibaba and Tencent on the cap table, as revenue ran from a $100M annualized run-rate in March to $200M in April. Of the pure-play labs, Moonshot has best converted benchmark leadership into commercial momentum.

ByteDance — scale, media, and the cloud beneath it

ByteDance is the platform counterweight to the open-weight labs, and — at a ~$480B secondary-market valuation (General Atlantic has tested $550B) — by far the most valuable of them. Its Doubao assistant reached 345M monthly active users and 120 trillion daily token calls, and its Volcano Engine cloud lifted its model-as-a-service revenue target to ¥15B (~$2.2B). ByteDance keeps its best models proprietary and behind APIs — the opposite of Qwen — because it monetizes attention and cloud, not weights, and it is far enough along on a self-designed inference chip (targeting 100k+ units in 2026) to hedge its Nvidia dependence. Its decisive lead, covered in Section 4, is in video.

The next tier — GLM, MiniMax, and the Xiaomi surprise

Three more names matter for anyone sizing the supply base. Zhipu (Z.ai), a 2019 Tsinghua spin-off now listed in Hong Kong at ~$56B, shipped GLM-5.2 — a 744B-parameter model that leads the Artificial Analysis Intelligence Index among open weights in June 2026, and is the strongest of the cohort in enterprise and government deployments. MiniMax (founded 2021 by ex-SenseTime researcher Yan Junjie; HK-listed ~$33B) pairs the M-series LLMs with its Hailuo video/audio models, straddling text and media. And the genuine surprise is Xiaomi: its MiMo models, an afterthought to a phone-and-EV company, are by several weekly counts the single most-used vendor on OpenRouter (~21% of tokens, ~22% of coding traffic). That a handset maker can lead global open-model usage is the strongest evidence that distribution and price, not research prestige, now decide share.

Chips and Supply: Huawei silicon, an HBM ceiling, and the offshore workaround

The software story above runs on hardware these labs increasingly cannot buy at will. China’s compute supply now rests on three legs: a maturing domestic accelerator in Huawei’s Ascend line, a still-open but narrowing channel to Nvidia silicon, and — until five weeks ago — a large offshore rental market in Southeast Asia. Each leg has a different fragility.

Huawei Ascend: competitive at the rack, constrained at the wafer’s edge

Huawei’s CloudMatrix 384 (CM384) is the clearest evidence that China can build at the system level. It lashes 384 Ascend 910C dies into a single super-node delivering roughly 300 PFLOPs of dense BF16 — close to 2× the compute of Nvidia’s GB200 NVL72 — by trading power efficiency for brute scale. At the rack, China has an answer. The constraint sits one layer down. Per SemiAnalysis, SMIC’s logic capacity is no longer the binding limit; high-bandwidth memory is. Domestic HBM from CXMT is projected at ~2.0M stacks in 2026, enough to finish only ~250,000–300,000 910C-class packages.

For investors, the takeaway is that China’s accelerator output is small relative to demand — a few hundred thousand high-end packages against domestic compute appetite measured in millions of GPU-equivalents — and that the leverage point for U.S. policy has moved from logic (SMIC, lithography) to memory (HBM, CXMT/Samsung/SK Hynix). The 910C is also less power-efficient than Nvidia’s parts, so China is spending its scarce energy and capital budget less efficiently per useful FLOP. That gap is closing, but it is real and it is the reason the offshore-Nvidia channel mattered so much.

The capex arms race

Constrained supply has not slowed spending — it has accelerated it, as the platform owners race to lock up whatever compute they can. Alibaba lifted full-year capex to ¥126.1B (≈ $17.5B), up from ¥84.3B. ByteDance is the more aggressive spender, raising 2026 capex past ¥200B (≈ $28–29B), with roughly half historically earmarked for AI chips — and some reports put its ceiling materially higher.

The offshore-compute loophole — opened, exploited, and just closed

Because the best Nvidia parts were export-restricted and Huawei output was capped, Chinese labs did the rational thing: they rented frontier GPUs offshore. For roughly a year, a China-parented company could stand up — or rent from — a data center in Singapore, Malaysia, or Indonesia and run banned Blackwell-class silicon legally, because the hardware never shipped to the mainland. ByteDance reportedly secured access to a 36,000-GPU Blackwell cluster through a Malaysian cloud operator, a deal Nvidia confirmed as compliant. That channel narrowed sharply on May 31, 2026, when BIS extended licensing to any buyer whose ultimate parent sits in China or Macau, regardless of where the subsidiary is registered.

The net picture on compute: China can build competitive systems but cannot yet make enough of the memory that finishes them, the cheapest path to frontier silicon (offshore rental) just narrowed, and the policy lever has migrated to HBM. Expect three responses — a domestic HBM build-out (CXMT capacity expansion), continued Ascend system scaling despite efficiency penalties, and more inference pushed onto the very open models from Section 2, which are cheap enough to run on constrained hardware. Scarcity of training compute is precisely why the open, efficient model tier is a strategic asset, not just a commercial one.

ByteDance and the video frontier: Not catch up — it is ahead

In text and code, the Chinese labs win on price and distribution while the U.S. frontier still sets the quality ceiling. In video and media generation, that ceiling is now Chinese. The shift is partly capability and partly economics: video models are absurdly expensive to serve, and the firms that win are the ones that can subsidize generation against another business — ads, short-form, a billion-user app. ByteDance has exactly that, and it has used it to take the lead.

ByteDance’s Seedance 2.0 (February 12, 2026) is a unified multimodal audio-video model: text, image, audio, and video in; synchronized video plus dual-channel stereo audio out, in a single forward pass, with native multi-shot inside one 15-second render. It topped Artificial Analysis’s Video Arena for both text-to-video and image-to-video from launch, and remains #1 on the with-audio board. Its only real challenger is also Chinese — Alibaba’s HappyHorse-1.0 — with Google’s Veo 3.1 holding third and Kuaishou’s Kling 3.0 placing four entries in the top ten.

Why the economics favor ByteDance, not OpenAI

The clearest proof that media generation is an economics problem, not just a quality problem, is Sora’s collapse. OpenAI’s consumer video product reportedly burned ~$15M per day in compute against ~$2.1M in total lifetime revenue — a day of cost exceeding all the revenue it ever earned by roughly 7×. OpenAI deprecated Sora in April 2026. ByteDance runs the same workload at the same scale, but against a short-video advertising machine that already monetizes attention. Seedance is not a standalone P&L; it is a feature of Doubao, Jimeng (Dreamina), Volcano Engine Ark, and BytePlus.

That structural subsidy is why ByteDance can keep a video model at the frontier while a pure-play cannot. It also explains ByteDance’s monetization posture: keep the models proprietary, expose them through APIs and apps, and capture value at the consumer and cloud layers. Volcano Engine raised its model-as-a-service revenue target to ¥15B on the strength of Seedance demand, and Doubao’s reach gives every model instant distribution.

Implications: Where margin survives when the model layer is free

The strategic conclusion is not “China is winning AI.” It is narrower and more useful: the open model layer has been commoditized, and China supplies it. That repricing is permanent, and it relocates where value accrues. For an infrastructure investor, five implications follow.

1. Weights are a funnel, not a moat. Qwen’s billion downloads are a customer-acquisition expense for Alibaba Cloud, not a product. Any thesis that underwrites a standalone open-weight model business as the durable asset is mispriced. The defensible positions are distribution (ByteDance, Alibaba), cost structure (DeepSeek), release cadence (Moonshot), and the cloud/inference layer that monetizes all of them. Margin lives above and below the model, not in it.

2. The binding constraint is memory, and so is the policy lever. China can design competitive systems (CM384) and has adequate logic capacity (SMIC), but domestic HBM caps 2026 high-end output near 250k–300k packages. That makes HBM the single highest-leverage node in the entire stack — for Beijing’s industrial policy, for Washington’s controls, and for investors. Watch CXMT’s ramp, Samsung/SK Hynix’s China exposure, and any HBM-specific export action; those move the real frontier more than any model release.

3. Scarcity of training compute is exactly why efficient open models are strategic. When you cannot buy unlimited frontier silicon, a model that delivers near-frontier quality at a tenth of the inference cost is not just commercially attractive — it is a national hedge. The cheapness that looks like a race to the bottom is, from Beijing’s side, a way to extract maximum useful intelligence from constrained hardware. Expect more inference pushed onto small, efficient, open models and more domestic Ascend deployment despite its efficiency penalty.

4. In media, back the distribution, not the generator. Sora’s shutdown is the cleanest data point of the year: a frontier-quality video model with no attached business is uninvestable at current unit economics. The winners are models welded to a consumer or ad platform that can subsidize generation — ByteDance first, Kuaishou and Alibaba behind it. Standalone video pure-plays are structurally short of the cash flow needed to stay at the frontier; the durable plays are the platforms and the inference infrastructure underneath them.

5. The DeepSeek panic is the template: price the disruption to the right layer. The market’s January 2025 error was not believing China would disrupt — it was putting the disruption on silicon when it belonged on model-vendor share. Efficiency gains flowed down to compute (Nvidia at $5.14T) and up to distribution; they were competed away in the middle, where the open Chinese labs now sit. That is also the risk to the labs themselves: at a combined ~$160B, the pure-plays occupy the thinnest-margin layer in the stack, and their equity value depends entirely on converting free weights into something that isn’t free — Alibaba’s cloud pull, ByteDance’s consumer surface, Moonshot’s enterprise ARR. For a U.S. investor, the cleanest hedge against Chinese model-share gains is to own the compute and memory layers that every one of those tokens still runs on.

What to watch over the next 12–18 months

• HBM: CXMT capacity beyond ~2M stacks; any HBM-specific U.S. control. The output ceiling moves with this number.

• Offshore enforcement: whether the May 31 BIS parent-company rule is actually enforced against Singapore/Malaysia/Gulf neoclouds, or merely announced.

• Frontier gap: whether DeepSeek/Kimi close the remaining quality gap to GPT-5.5 / Claude Opus on agentic tasks, or plateau just below it.

• Monetization: Alibaba Cloud AI run-rate (past ~$5B) and ByteDance Volcano MaaS revenue (¥15B target) — the proof that open weights convert to cloud margin.

• U.S. response: whether a credibly open, efficient U.S. model re-enters the Hugging Face top tier, or the open layer stays Chinese by default.

Summary

As of June 2026, China supplies the majority of the world’s open-model tokens, sets the price of intelligence at the open tier, and leads outright in video generation. It does this from a constrained compute base — competitive systems, capped memory, a just-closed offshore channel — which is precisely why efficiency and openness are central to its strategy rather than incidental to it. The U.S. frontier still defines the quality ceiling and captures most industry revenue, but it no longer controls the default. The contest has moved from “who builds the best model” to “who controls the layers that still carry margin”: memory and interconnect below the model, distribution and cloud above it. That is where the next five years of returns — and the next round of export policy — will be decided.

Sources

1. OpenRouter — State of AI: 100T-Token LLM Usage Study and LLM Rankings.

2. KuCoin / Global Times — OpenRouter: 61% of token consumption by Chinese models; Chinese models take top six usage spots.

3. digitalapplied — OpenRouter Rankings April 2026; Chinese AI Models Q2 2026 Landscape.

4. Hugging Face — State of Open Source, Spring 2026; One Year Since the DeepSeek Moment.

5. Pandaily — Qwen surpasses 1 billion downloads; getaibook — Chinese open models overtake U.S. on the Hub.

6. DeepSeek — API Pricing; verdent.ai — V4 Pricing & Migration (2026); sitepoint — DeepSeek V4 Released.

7. Moonshot AI — Kimi K2.6; miraflow — K2.6 ties GPT-5.5 on coding; DevOps.com — Kimi K2.7-Code.

8. SemiAnalysis — Huawei Ascend Production Ramp (HBM bottleneck); CloudMatrix 384; Tom’s Hardware — Ascend ecosystem scales.

9. Alibaba — FY2026 6-K; CIW — Alibaba Cloud Q1 2026. ByteDance capex — EqualOcean, SCMP.

10. Export controls / neoclouds — Tom’s Hardware — ByteDance 36k Blackwell via Malaysia; Asia Times — SE Asia data-center crackdown; Indoneo — BIS parent-company rule.

11. ByteDance video / Doubao — ByteDance Seed models; sitepoint — Seedance 2.0; kr-asia (Volcano MaaS target); Sacra — ByteDance overview. Video market — digitalapplied: AI video after Sora; Bloomberg.

12. DeepSeek panic / Nvidia — CNBC — Nvidia sheds ~$600B, biggest one-day loss; IG — why NVDA fell 17%; MacroTrends — NVDA market cap ($5.14T, Jun 2026).

13. Vendor share / Meta decline — stockalarm — OpenRouter rankings investor analysis (Chinese 46–56%); cryptobriefing — U.S. startups shifting to Chinese LLMs.

14. Valuations — TechCrunch — Moonshot $2B at $20B; SCMP — DeepSeek first raise; Bloomberg — ByteDance ~$480B; companiesmarketcap — Chinese company market caps.

The dominant narrative of the last software cycle was that value migrates up the stack — toward the application, the workflow, the system of record, the seat you sell. That was true for SaaS, where the marginal cost of serving one more user approached zero and durable 75–90% gross margins accrued to whoever owned the interface.

AI inverts it. The center of gravity is moving down the stack, toward the layers with the highest capital intensity, the deepest physical constraints, and the strongest control points: semiconductors and compute, data platforms, open models and the inference engines that run them, and a narrow premium frontier tier. The application layer does not disappear — but it thins. Its margins compress as inference becomes a variable cost of goods, and its multiples bifurcate: the model and the data re-rate upward, the wrapper does not.

Stated plainly: the future is a very large number of open-source models (with inference platforms to run and optimize them), a smaller set of premium models, data platforms, and the semiconductors to power all of it — sitting beneath a thinner application software layer than the SaaS era trained us to expect.

The rotation reverses

This is heretical to how private capital has thought for twenty years. After the dotcom crash, investors fled capital-intensive, cyclical semiconductors and rotated decisively into asset-light cloud and SaaS — 80%+ gross margins, near-zero marginal cost, recurring revenue, and capital efficiency that compounded into the highest multiples in the market. “Software is eating the world” became the organizing thesis of an entire generation of funds; the LP and GP base reorganized around application software, and the semiconductor industry was treated as a low-multiple, cyclical afterthought.

AI reverses the rotation. Capital is flowing back down into the physical, capital-intensive layers the industry spent two decades fleeing — silicon, packaging, power, data infrastructure, and the inference plumbing on top. The most valuable franchise in technology is, again, a chip company; the fastest-growing infrastructure businesses ever recorded are inference and data platforms. The asset-light application layer — the destination of the last rotation — now carries the worst margin structure in the stack. Twenty years of muscle memory points the wrong way.

Framework: value has gravity

Treat the AI stack as five layers and ask one question of each: how easily can the next dollar of value be substituted away? Value, like mass, accumulates where it cannot easily move. Four properties pull value down the stack and hold it there.

1. Capital intensity. The base layers cost tens of billions to build, continuously. The four largest hyperscalers are spending at a $500B+ annualized pace on capex, heading toward $600B+ for 2026. Capital is a moat when it is both enormous and perpetually required.

2. Physical bottlenecks. Advanced packaging (CoWoS), high-bandwidth memory, and power are hard limits that money cannot relax on a quarterly cadence. Scarcity accrues to whoever controls the constraint.

3. Switching cost and inertia. Data has gravity in the original sense: every pipeline, table, and model trained in place raises the cost of leaving. Open weights are the opposite — engineered to be portable — which is exactly why value at that layer shifts from the model to the platform that serves it.

4. Replacement risk at the surface. Generic application logic is the most substitutable thing in the stack, because the model can increasingly perform it directly.

The rest of this piece walks the layers from the base up, with the numbers.

Layer 1 — Semiconductors and compute: the densest mass

This is the heaviest layer, and it is getting heavier. In its most recent quarter — ended April 2026 — NVIDIA’s data-center revenue hit $75.2B, up 92% year over year, at a ~75% gross margin: roughly a $300B annualized run-rate, with the current quarter guided to $91B in total revenue. NVIDIA holds about 80% of AI accelerators by revenue. Those are not the economics of a competitive commodity market; they are the economics of a control point.

The demand beneath it is a capex supercycle without parallel in enterprise technology. In Q1 2026 alone, the four largest hyperscalers spent roughly $131B on capex — Amazon $44.2B, Alphabet $35.7B, Microsoft $30.9B, Meta $19.8B — a ~$525B annualized pace, still climbing toward $600B+ of full-year 2026 guidance. Goldman Sachs models $5.3T of cumulative hyperscaler capex from 2025 through 2030.

And it is not only compute. The entire semiconductor industry is in a supercycle: global revenue hit $793B in 2025 (+21% YoY) and is on track to approach $1 trillion in 2026 (WSTS ~$975B, +25%; Gartner higher still), with AI now ~30% of all semiconductor revenue and headed past 50% by 2029. The demand splits into three buckets, and all three are supply-constrained. Compute — logic, GPUs, and accelerators — is the largest: AI accelerators alone scale from ~$80B in 2024 to $280B+ by 2029. Memory is the fastest-moving: HBM demand grew ~130% in 2025 and another ~70% in 2026, conventional DRAM contract prices jumped ~90% quarter-over-quarter in Q1 2026, and AI data centers are absorbing roughly 70% of high-end DRAM. Storage is quietly tightening too: the NAND market reaches ~$65B in 2026, one in five bits now feeds AI, and demand (+20–22%) is outrunning supply (+15–17%) into 2027. When a single end-market reprices DRAM 90% in a quarter and books advanced packaging two years out, the constraint — and the margin — has moved into the physical layer.

Two features make this layer durable rather than merely large. First, the binding constraint has moved from chip design to packaging and power. TSMC’s CoWoS capacity — about 70k wafers/month in 2025, rising toward ~110k in 2026 — is effectively sold out, and NVIDIA has reserved the majority of supply into 2027. Second, more than 10 GW of new AI data-center capacity was announced in 2025, against a packaging base that can support on the order of 18 GW. When the bottleneck is physical, capital intensity is a feature for incumbents and a wall for entrants.

The replacement risk here is real but slow: hyperscaler custom silicon (Google TPU, AWS Trainium, Broadcom ASICs) is growing into a $200B+ accelerator TAM. It compresses NVIDIA’s share over time; it does not move value out of the layer. The mass stays at the bottom.

The real risk at this layer is returns, not revenue. At a $500B-plus annual capex run-rate against accelerators that depreciate over few year useful lives, margin durability depends on utilization staying high enough to earn a return on the installed base. That — not competition — is the single largest swing factor beneath the entire stack.

Layer 2 — Open models and inference: abundance at the model, value at the platform

The single most important quantitative story in AI over the past 24 months is convergence. Per the Stanford AI Index, the gap between the best closed model and the best open-weight model on Chatbot Arena fell from 8.04% in January 2024 to 1.70% in February 2025. On MMLU, the 2023 gap of ~17.5 points has effectively closed. DeepSeek R1 showed that frontier-class reasoning does not require nine-figure training budgets; Qwen overtook Llama as the most-downloaded model family on Hugging Face.

The supply side is now effectively infinite. Hugging Face hosts more than 2.2M models, with the catalog doubling in roughly 24 months. When a capability exists in thousands of near-equivalent open variants, the weights themselves stop being the scarce asset. The scarce asset becomes the ability to run them well — throughput, latency, cost per token, routing, caching, fine-tuning, evaluation, and guardrails.

There is a market-structure wrinkle investors should not miss: the open-weight frontier is increasingly non-US. Chinese developers accounted for 17.1% of Hugging Face downloads from August 2024 to August 2025, edging out the US at 15.8%, and Qwen displaced Llama as the most-downloaded family. The commoditizing layer is also the layer least controlled by US incumbents — a supply-chain and policy variable, not only an economic one.

Linux did not capture the value of Linux; Red Hat and the cloud did. The same pattern now applies to open weights — value accrues to the platform that serves them, not the model itself.

This is why the durable business at this layer is the inference and optimization platform, not the open model. The platform sits between abundant open supply and enterprise demand, and monetizes the one thing that does not commoditize: making a portable artifact perform reliably and cheaply in production.

The market is proving this in real time, at a velocity with no precedent in infrastructure software. Fireworks AI disclosed it crossed $800M in annualized revenue in roughly three years — up over 4× year over year. Together AI is rumored to be at a similar scale; Baseten reached ~$600M, up >5X year over year. None of these are model labs — they are the toll roads that make open weights usable in production. When the model itself is free, the company that serves it fastest and cheapest captures the spend.

The math is why. Consider a team serving 50B output tokens a month — a mid-size AI feature. At frontier-API output pricing of ~$10–15 per million tokens, that runs ~$500–750K a month; the same workload on an open model (Llama, Qwen, DeepSeek) served through an inference platform at ~$0.40–1.00 per million costs ~$20–50K — a 90%+ reduction. Below a quality threshold that keeps falling, open-plus-inference wins on cost by an order of magnitude, and the platform that delivers it keeps a slice of the savings as margin. That is the whole business. (Illustrative; actual costs vary with model, context length, and utilization.)

Layer 3 — Premium models: a smaller, real, defensible tier

Convergence compresses the average but not the frontier. A premium tier persists wherever the marginal point of capability — complex agentic tasks, long-horizon reliability, production coding, safety — is worth paying for. Revenue proves it: Anthropic reported ~$47B annualized by May 2026 (from ~$1B at end-2024) and is tracking toward an estimated ~$100B run-rate by year-end; OpenAI — ~$6B at end-2024, ~$20B at end-2025, ~$30B in mid-2026 — is on a path to an estimated ~$60B. More than 1,000 enterprises spend over $1M/year with Anthropic alone.

(Both labs report on different revenue bases, and year-end 2026 figures are estimates; treat the absolute numbers as directional and the trajectory as not.)

The strategic point for investors: this is a narrower layer than the 2023 consensus assumed. Open weights have taken the floor; the premium tier monetizes the ceiling. The defensible position is the frontier plus the distribution to reach enterprises — not “a good model,” which is now free and plentiful.

Layer 4 — Data platforms: the gravity well that compounds

If semiconductors are the densest layer, data platforms are the stickiest. This is “data gravity” in its literal form: the cost of moving away rises with every pipeline, table, and ontology built in place.

The numbers show the layer capturing the AI workload directly — and the market paying premium multiples for it. Databricks crossed a $6.9B annualized run-rate growing ~80%, with AI products already ~26% of revenue, valued around $170B privately. Palantir — the operational layer that turns enterprise data into governed decisions — grew revenue 85% year over year in Q1 2026 (its fastest ever) to a ~$6.5B run-rate, with US commercial revenue up 133% and a Rule of 40 of 145%. At a ~$350B market cap, the market pays Palantir a foundation-model-class multiple — roughly 50× revenue — for owning the decision loop that sits on top of data.

Why does value hold here while it leaks at the model layer? Because data is the one input that is both non-substitutable and accretive. Models converge; a company’s proprietary data — and the workflows and ontology built on it — does not. As the model layer commoditizes, the platform that owns the data and the decision loop becomes the durable point of leverage. Note the inversion this implies for the application thesis: the layers commanding the highest multiples — Palantir at ~50× revenue, foundation models at 25–50× — are precisely the ones fused to proprietary data, not the generic software surface.

Layer 5 — The application layer: thinner than the SaaS playbook assumes

Now the top. The thesis is not that applications die; it is that the generic application surface thins, in two measurable ways.

First, margins. Traditional SaaS runs 75–90% gross margins because marginal serving cost approaches zero. AI-native applications run 50–60%, because every query re-runs a model. By ICONIQ’s 2026 data, inference averages ~23% of revenue at scaling AI companies — and, critically, it does not fall with scale the way SaaS COGS did. The thin-wrapper extreme can sit at 25% gross margin. Inference is a structural, variable cost of goods.

Second, multiples. The market is already pricing the bifurcation. Foundation-model companies trade at 25–50× revenue; AI-native platforms at 25–30×; traditional SaaS has compressed to a ~6.7× median (from an 18.6× peak in 2021); and “AI wrappers” sit at 5–8× — at or below legacy SaaS. Value re-rates to the model and the data; it derates at the undifferentiated surface.

This does not mean “don’t invest in applications.” It means the bar moved. A durable application now needs something the model cannot absorb: a proprietary data loop, a system of record, a regulated workflow, real distribution, or outcome-based pricing that converts inference cost into margin. Absent that, the application is a thin layer of prompt-and-glue over a commoditizing model — and the multiple will say so.

The counter-thesis: what would make me wrong

Decisive does not mean unfalsifiable. Three developments could redistribute value back toward the top.

Apps capture the data loop. If the best applications accumulate proprietary interaction data fast enough, they become data platforms in their own right and inherit the gravity. This is the strongest bull case for the application layer, and the reason the system-of-record app is exempt from the thinning.

Vertical integration collapses layers. A premium-model lab that owns inference, builds agents, and sells the application directly could re-internalize value across three layers at once. That concentrates value rather than redistributing it upward — but it does erode the independent inference and app layers.

Inference commoditizes faster than expected. LLMflation — roughly a 10×/year decline in the cost of a unit of intelligence, with median price drops near 200×/year since early 2024 — cuts both ways. If serving becomes nearly free, AI-app gross margins recover toward SaaS norms and the margin half of the thinning thesis weakens. (The multiple half largely holds, because substitutability, not cost, drives it.)

Note the tension: the same cost collapse that pressures the semiconductor and inference layers on price is what could make the application layer’s margin problem temporary. I weight the structural argument above the cost-relief argument, because substitution risk at the surface persists even at zero inference cost.

Implications

For investors. Underwrite control points, not interfaces. The highest-conviction, lowest-substitution positions are: compute and the physical bottlenecks around it (packaging, power, memory); data platforms with proprietary accretion; and the inference/optimization layer that monetizes open-model abundance. In the premium-model tier, underwrite the frontier plus distribution, and size for a narrower-than-consensus winner set. At the application layer, require a data loop or a system of record before paying a software multiple — and assume 50–60% gross margins, not 80%+, until proven otherwise.

For founders. If you are building an application, your job is to manufacture gravity. Own a proprietary data loop, become a system of record, or price on outcomes so that falling inference cost flows to your margin rather than your customer’s savings. “Thin” describes the default, not a sentence.

For incumbents. The capital intensity of the base layers is a moat you can rent but not own. For most enterprises the defensible move is up at the data layer — keeping data, training, and serving in one place — rather than competing on compute or frontier models.

What would confirm — or break — this

Five measurable signals over the next four quarters will tell you whether the heavy stack is holding:

• Hyperscaler capex-to-revenue. If it keeps climbing past ~25–30% of revenue without denting margins, the base layer’s returns thesis holds; if ROIC cracks, the whole stack reprices.

• Inference-platform ARR. Fireworks, Together, and Baseten compounding toward ~$1–2B each confirms value is pooling at the inference layer, not the model.

• The open–closed gap. If it stays inside ~2 points on hard benchmarks (agentic, coding), commoditization of the model layer is locked in.

• AI-app gross margins. If scaling AI apps stay at 50–60% (inference not falling with scale), the thinning thesis holds; a recovery toward 75%+ would weaken it.

• DRAM and HBM pricing. Continued double-digit quarterly price increases mean the physical constraint — and the margin — is still at the base.

Summary: probability-weighted, 3–5 years

The recommendation is consistent across the high-probability branches: position at the base and the data layer, treat the premium-model tier as a narrow high-conviction bet, and hold applications to a higher bar — a data loop or a system of record — before underwriting them at software multiples. Gravity is doing the work. Invest where the mass is.

Sources: NVIDIA Q1 FY2027 results (May 2026); company Q1 2026 earnings (Amazon, Alphabet, Microsoft, Meta); Stanford HAI AI Index 2025; Hugging Face; Epoch AI; Andreessen Horowitz ("LLMflation"); Gartner and WSTS (semiconductor revenue); TrendForce and Counterpoint (DRAM/NAND/HBM); BofA; Databricks and Palantir disclosures; OpenAI, Anthropic, Fireworks AI, Together AI, Baseten (via Sacra / PYMNTS); ICONIQ State of AI 2026; Goldman Sachs; ValueAddVC; SaaS Capital. Private-company revenue run-rates are reported on differing bases; year-end 2026 model-lab figures are estimates.

Compute Express Link (CXL) is an open, cache-coherent interconnect that lets CPUs, accelerators, and memory devices share one address space over the PCIe physical layer. It exists because memory stopped scaling at the rate compute does, and because the unit of AI infrastructure spend has shifted from FLOPS to bytes held in fast memory.

CXL is the mechanism for decoupling memory capacity from the CPU socket it used to be soldered next to — turning memory into a pooled, fabric-attached resource instead of a fixed, stranded one. Seven years after its 2019 launch, it has gone from a side project to the default answer for memory disaggregation: all three big DRAM makers build CXL memory modules, every major CPU exposes CXL root ports, and the spec hit its fourth major revision in November 2025. The timing is not incidental. With DRAM prices up roughly 90% quarter-over-quarter in Q1 2026 and HBM crowding conventional DRAM off the wafer, the economic case for reclaiming stranded memory just went from theoretical to urgent.

Key Points

What is CXL in simple terms?

Compute Express Link is an open standard that lets a CPU treat memory attached over a PCIe cable or slot almost like its own local RAM — readable with ordinary load/store instructions, kept cache-coherent — so capacity and bandwidth can be added, pooled, and shared instead of being fixed to each socket.

What problem does CXL solve?

The memory wall. Compute has scaled ~600× faster than memory bandwidth over two decades, and AI inference (especially KV cache) consumes memory per user. CXL adds a cheaper, poolable memory tier and recovers DRAM that would otherwise sit stranded.

Is CXL used to connect GPUs to each other?

No. GPU-to-GPU scale-up is the domain of NVLink and UALink. CXL targets CPU-to-memory expansion, pooling, and coherency — the two are complementary, not competing.

Who are the main CXL companies?

Astera Labs (Leo controllers), Marvell (Structera controllers and switch), Credo (SerDes/retimers), Micron, Samsung and SK hynix (memory media), and Penguin Solutions and MemVerge (systems and software).

The Bottleneck: Compute outran memory by 600x

The mechanics start with a scaling divergence. Over the past two decades, peak compute (FLOPS) has grown roughly 3.0× every two years — about 60,000× cumulatively — while DRAM bandwidth grew ~1.6× per two years (~100×) and interconnect bandwidth ~1.4× (~30×). That is a ~600× gap between how fast processors can compute and how fast memory can feed them. This is the “memory wall,” and it is the single largest cost driver in LLM inference today.

As I detailed in Why KV Cache and Memory Drive AI Economics, a 128K-token conversation on a 70B-parameter model can consume ~40GB of HBM per concurrent user — memory that scales linearly with users and context length, unlike the fixed footprint of model weights. HBM is the fastest tier, but it is capacity-constrained and priced at a premium: a single HBM3E stack runs $60–100 against $5–10 for an equivalent slice of DDR5, and HBM consumes roughly 3× the wafer area per gigabyte. The result in 2026 is a genuine DRAM shortage. CXL exists to put a coherent, low-latency tier between scarce HBM and a slow storage hop — and to recover the DRAM that is already installed but stranded.

The Demand Driver: Context windows are the underappreciated trend

The demand curve underneath the memory wall is context length. Two years ago GPT-4 shipped with an 8K-token window; today Gemini runs at 2M and Claude- and GPT-5-class models sit at 1M+, with roughly thirteen frontier models now shipping million-token windows. Micron EVP Manish Bhatia told JPMorgan in 2026 that context windows are expanding ~30× per year. Every token in a window generates KV-cache data that must live in memory during inference, so a single 1M-token session can consume hundreds of gigabytes of KV cache per user — and agentic workflows, where one request fans out into many chained model calls each carrying its own context, make that growth combinatorial rather than linear.

This is the flywheel: longer context → larger KV caches → more HBM and DRAM consumed per session (a direct tailwind for Micron and the NAND makers), and, when the cache overflows on-package memory, a forced spill outward to disaggregated CXL memory (where Penguin Solutions and the controller vendors monetize). A GPU’s on-package HBM is physically bounded by interposer area and stack height to roughly 80–192 GB; once the working set exceeds that, the only question is which tier catches the overflow.

How CXL works, in plain terms

Strip away the jargon and CXL does one thing: it lets a CPU reach extra memory sitting on a PCIe link and use it almost like its own RAM. It does this with three sub-protocols that share a single physical wire. CXL.io handles discovery and configuration — the housekeeping, functionally identical to PCIe. CXL.cache lets a device read the host’s memory while staying coherent (kept in sync). And CXL.mem — the important one — lets the host read the device’s memory with ordinary load/store instructions, no storage driver in the path.

CXL.io → find & set up the device · like PCIe
CXL.cache → device reads the CPU's memory, in sync · coherent
CXL.mem → CPU reads the device's memory, load/store · the key one

The practical consequence is the whole point of the technology: software treats CXL-attached DRAM the same way it treats local DRAM, just ~200 nanoseconds further away (about a NUMA hop), versus the ~100 microseconds it would take to reach an SSD. Around this, the spec defines three device types: Type 1 (an accelerator or SmartNIC with no memory of its own), Type 2 (a GPU or FPGA that has its own memory and also shares the host’s), and Type 3 (a plain memory expander or pool — the workhorse deployed against the memory wall).

Memory semantics, not I/O semantics, is what makes CXL a memory technology rather than just a faster peripheral bus.

The Roadmap: From expansion slot to memory fabric

Each spec revision pushed CXL further from “a card in a server” toward “composable memory across a rack.” CXL 1.1 (2019) did direct-attached expansion on a PCIe 5.0 PHY at 32 GT/s. CXL 2.0 (2020) added switching and memory pooling, plus hot-plug, QoS, and link encryption. CXL 3.0 (2022) moved to a PCIe 6.0 PHY at 64 GT/s with PAM-4 signaling, doubled bandwidth, and introduced fabrics: multi-level switching, memory sharing across hosts, peer-to-peer DMA, and addressing for up to 4,096 nodes. CXL 3.1 (2023) added multi-tier switching and trusted-execution security. And CXL 4.0, released in November 2025, doubled per-lane bandwidth again on a PCIe 7.0 PHY at 128 GT/s, with bundled ports and multi-rack pooling. The throughline: every generation widened the blast radius of a single memory allocation, from one motherboard to an entire rack.

Who is behind CXL — and what CXL is not

CXL is governed by the CXL Consortium, founded in 2019 by Alibaba, AMD, Cisco, Dell EMC, Meta, Google, HPE, Huawei, Intel, and Microsoft — a roster that signals hyperscaler-driven standardization rather than one vendor’s proprietary play. Critically, the Consortium then won the interconnect wars: it absorbed Gen-Z (2021), OpenCAPI (2022), and the CCIX effort, consolidating a fragmented landscape into one standard with one roadmap. That removed the standards-war risk that historically discourages capex on new interconnect silicon, which is why every major CPU — Intel Xeon, AMD EPYC, and the Arm server ecosystem — now ships CXL root-port support natively, and why AMD, Nvidia, and Samsung all sit on the board.

One clarification for the technical reader: CXL is not the GPU-to-GPU scale-up fabric. Its bandwidth and radix are wrong for that job, which belongs to NVLink (proprietary) and UALink (open). CXL won the CPU-to-memory layer; it never seriously contested the accelerator-to-accelerator layer. The two are complementary — and the most telling signal is that even Nvidia supports CXL memory expansion on Grace and Blackwell while building NVLink for scale-up.

Key Players

Connectivity silicon: Astera, Credo, Marvell

Astera Labs (ALAB) is the highest-profile pure-play. It IPO’d in March 2024 at more than 50× trailing revenue — I broke down the filing in Astera Labs IPO: S-1 Teardown, when Aries retimers drove 90%+ of revenue. The scale since is the story: full-year 2025 revenue hit $852.5M (+115%), and the most recent quarter — Q1 FY2026 (reported May 2026) — set a record at $308.4M, up 93% year-over-year and 14% sequentially, at a 76.4% non-GAAP gross margin and a 36.2% non-GAAP operating margin, with PCIe Gen 6 already more than a third of revenue. Leo, its CXL smart-memory-controller line (CXL 2.0, up to 2TB per controller), is the strategic bet — it powers Microsoft Azure’s M-series VMs in the industry’s first announced CXL-attached memory deployment, and in distributed inference with Penguin Solutions it has shown 75% higher GPU utilization and 2× inference throughput, plus 40% faster time-to-insights and up to 3× concurrent LLM instances for RAG. Leo plugs into vLLM, TensorRT-LLM, and SGLang with no application changes.

Credo (CRDO) occupies the adjacent retimer and SerDes layer. The growth has been violent: fiscal 2026 revenue hit $1.3B, up 206% year-over-year, with a record Q4 of $437M (+157% YoY) at ~68% gross margins — and the stock now carries a roughly $50B market cap, four times the level when I first covered it in From Niche to Necessary: Credo’s Quiet Domination of AI Interconnects. Its retimers support PCIe 5.0/6.0/7.0 and CXL interchangeably, so it captures CXL-attach revenue regardless of which controller or memory vendor a hyperscaler standardizes on. Marvell, meanwhile, is building the switch and fabric layer underneath both. Its Structera A near-memory accelerators (16 Arm Neoverse V2 cores, up to 200 GB/s and 4TB) and Structera X expansion controllers were the first with four memory channels and inline compression on 5nm; then in March 2026 it announced the Structera S 30260, a 260-lane CXL 3.0 switch delivering up to 4 TB/s of aggregate bandwidth for rack-level pooling, sampling Q3 2026 — built on Marvell’s $540M acquisition of XConn, closed February 2026, the largest deal in CXL switching to date.

Memory and systems: Micron, Samsung, SK hynix, Penguin

The memory makers supply the capacity the connectivity silicon pools, and CXL is their channel into sockets that are otherwise capacity-capped. Micron (MU) is the cleanest read on the demand: fiscal Q2 2026 revenue hit $23.86B, up 196% year-over-year, at a ~74% gross margin, with Q3 guided near 81% and management describing HBM and high-capacity DRAM as effectively sold out ahead of supply. Its CZ120 CXL modules ship in 128GB and 256GB capacities over PCIe Gen5 x8 at ~36 GB/s — within a hair of a DDR5-4800 RDIMM’s 38.4 GB/s. Samsung’s CMM-D line reached an industry-first 512GB module, and SK hynix ships its 96GB CMM-DDR5 with pooling support. None of them needs CXL to sell DRAM, but all of them need a way to sell more DRAM per core while HBM eats fab allocation — and NAND makers (Micron again, plus SanDisk) pick up the colder KV-cache tiers below CXL.

At the top of the stack, Penguin Solutions (PENG) is the clearest evidence that CXL demand is now inference-driven. In March 2026 it shipped what it calls the industry’s first production-ready CXL-based KV cache server, the MemoryAI appliance — 3TB of DDR5 plus up to eight 1TB CXL add-in cards for up to 11TB total — purpose-built to offload KV cache from GPU HBM, with roughly 10× faster access than NVMe-based approaches and out-of-the-box compatibility with NVIDIA’s Dynamo framework. Penguin frames the workload bluntly: inference is roughly 30% compute and 70% memory. The pull is showing up in results — Q2 FY2026 integrated-memory sales rose 63% year-over-year to $172M, and the company raised full-year guidance to ~12% growth.

Why CXL Now?: The 2026 memory squeeze