Thesis

The binding constraint on frontier AI has shifted. Through 2023 it was accelerator supply. By 2026 it is the interconnect — and “interconnect” now spans three distinct distance scales at once: chip-to-memory (millimeters), GPU-to-GPU within a rack (meters), and rack-to-rack across a data center hall (tens to hundreds of meters). The fabric moving activations, gradients, KV-cache traffic, and now memory bandwidth itself between tens of thousands of accelerators has to keep all three scales fed simultaneously, fast enough that the silicon never idles waiting on data.

Copper, the default interconnect medium for sixty years of computing, has hit a hard physical limit at the data rates AI clusters now require. A growing field of replacement technologies is racing to fill the gap — silicon photonics, VCSEL-based near-package optics, THz radio-over-wire, and early-stage microLED interconnects among them — and 2026 is the year several of them moved from roadmap slide to shipping product.

Two approaches anchor this piece because they represent the clearest structural bets being made with real capital today: silicon photonics, which converts data to light and is now central to Nvidia’s and Broadcom’s switch roadmaps, and THz radio-over-wire, an approach pioneered by the startup AttoTude (a Wing portfolio company) that drives terahertz-frequency radio signals directly through wire, skipping optical conversion altogether. These are not rivals fighting over a single socket. They sit at different points on the cost, distance, and power curve, and the more durable read — reinforced by how Marvell, Broadcom, and Nvidia are each shipping multiple interconnect formats side by side inside the same switch families — is that several of these approaches will coexist in the AI data center of 2028–2030, often inside the same rack.

The scale of the prize is enormous regardless of which technologies win which sockets. The five largest hyperscalers are on track to spend roughly $725 billion on AI infrastructure in 2026 alone — nearly triple the $256 billion they deployed in 2024 — and networking and interconnect have become a double-digit percentage of total cluster cost rather than a rounding error. Nvidia has put a number on how seriously it takes this: Nvidia has directed roughly $4 billion of capital specifically into optical component suppliers — split into $2 billion stakes in Lumentum and Coherent, paired with multibillion-dollar purchase commitments, future capacity-access rights, and funding for new U.S.-based fabrication facilities, a direct, dollar-denominated bet that the interconnect layer — not the GPU itself — is now the gating factor on how fast it can ship usable AI compute. McKinsey, separately, estimates that EML laser chips will run 40–60% undersupplied through 2027.

The companies that control the bottlenecks in this build-out — manufacturing capacity, component and architecture IP, and systems-level co-design with the largest compute buyers — are re-rating from cyclical component suppliers into structural infrastructure plays. That re-rating is visible in the numbers today, not just the roadmaps.

The numbers that frame the opportunity:

• Frontier training clusters in 2026 already run 100,000+ GPUs, with “AI factory” designs targeting millions — and copper’s practical reach collapses as speed rises: Marvell’s own data points put it at roughly 5 meters at 100 Gb/s per lane, ~2.5 meters at 200 Gb/s, and effectively unusable even within a rack at 400 Gb/s.

• Hyperscaler AI infrastructure spend is on pace for ~$725 billion in 2026, up from $256 billion in 2024 — and networking and optical interconnect, now a double-digit percentage of total cluster cost rather than a rounding error, represents on the order of $75–100 billion of that spend this year alone.

• Nvidia has committed roughly $4 billion directly into optical suppliers, and its Spectrum-X Photonics SN6800 switch reaches 409.6 Tb/s across 512 ports at 800 Gb/s each — with co-packaged optics cutting power consumption by up to 3.5x and improving resiliency 10x versus pluggable architectures.

• Marvell’s new Teralynx T100 — the industry’s first 102.4 Tb/s switch built specifically for AI data centers — runs on a 3nm process, draws under 1,000W, and delivers roughly 25% lower power than competing solutions, while shipping in pluggable, co-packaged-copper, and co-packaged-optics configurations side by side.

• The silicon photonics market is estimated at roughly $3–4 billion in 2025–2026, scaling to $10–29 billion by 2030–2034 (roughly a 27–30% CAGR across the major forecasts — Mordor Intelligence, MarketsandMarkets, Precedence Research, and Polaris Market Research all now cluster in the high-20s, with several houses revising estimates upward through 2026 as AI-specific demand outpaces the telecom- and consumer-driven base case those models were originally built on), while the broader next-generation AI interconnect market — spanning photonics, VCSEL/NPO, THz, and microLED approaches — could exceed $100 billion annually within the decade, according to industry estimates cited around OFC 2026.

• AttoTude’s THz radio-over-wire approach reports per-lane speeds up to 448 Gb/s and reach up to 40 meters directly over wire — without an optical conversion stage.

The Copper Wall: Why Electrons Stop Scaling

Copper interconnects move data by modulating voltage on a wire — a mechanism that degrades predictably, and non-linearly, as both frequency and distance rise. Marvell CEO Matt Murphy put concrete numbers on this at his Computex 2026 keynote: copper can carry a signal roughly 5 meters at 100 Gb/s per lane, but that reach falls to about 2.5 meters at 200 Gb/s, and at 400 Gb/s copper can no longer reliably make connections even within a single rack. Each time this threshold moves, Murphy argued, the number of links that must convert to optical increases by at least an order of magnitude — which is what is now driving what he called an “explosive” demand cycle across the optics supply chain, with Taiwan’s manufacturing base already expanding in response.

This is the “copper wall,” and it is a property of physics that worsens, not improves, as model and cluster sizes grow. Beyond roughly three meters of reach, attenuation and heat dissipation make copper a net negative versus the alternatives; at million-GPU scale, the share of total system power and latency consumed by electrical interconnect stops being an afterthought and becomes the dominant design variable.

Every replacement technology in this piece — optical, VCSEL-based, THz-over-wire, or microLED — exists because it sidesteps this specific failure mode, via different physical mechanisms. Photons don’t suffer resistive loss the way electrons do. THz-frequency radio signals can be engineered to traverse wire with substantially less of the attenuation that kills copper at lower frequencies. And the industry’s accelerating search for more than one way around this wall is itself the signal: when an entire supply chain starts hedging across several physical bets simultaneously, that is usually a sign the old approach is genuinely exhausted, not just temporarily expensive.

Why This Matters Specifically for Large Models

Transformer-based models are bottlenecked as much by communication as by computation. Mixture-of-experts architectures, tensor and pipeline parallelism, and disaggregated inference — separating prefill from decode across different hardware pools — all multiply east-west traffic between accelerators relative to the FLOPs actually executed. As clusters scale from thousands to hundreds of thousands of GPUs, the ratio of interconnect bandwidth to compute, not raw FLOPs, increasingly determines effective utilization.

Nvidia has made this an explicit architectural priority. Its Quantum-X InfiniBand switches (early 2026) and Spectrum-X Photonics Ethernet switches (second half of 2026, built on the Spectrum-6 ASIC) integrate co-packaged optics directly onto the switch silicon. The flagship SN6800 reaches 409.6 Tb/s across 512 ports at 800 Gb/s — bandwidth that would be physically and thermally impossible to deliver with discrete pluggable copper or pluggable optical modules at that density. Nvidia states CPO cuts power consumption by up to 3.5x and improves resiliency 10x versus conventional pluggable architectures, with its longer-range roadmap following TSMC’s COUPE packaging program toward 6.4 Tb/s at the board level.

Nvidia’s own architecture diagrams make the structural shift legible at the component level. A traditional pluggable switch routes signal from the switch ASIC, across the substrate, through a port cage, into an external optical transceiver — DSP, externally modulated laser, and all — needing eight separate lasers per 1.6 Tb/s of bandwidth. Spectrum-X Ethernet Photonics collapses that chain: the silicon photonics engine sits directly on the substrate beside the switch ASIC, driven by a continuous-wave laser, and the same 1.6 Tb/s now requires just two lasers — a quarter as many laser sources, connectors, and potential failure points, assembled in far fewer steps.

Crucially, this is no longer a future-tense story. Nvidia’s Vera Rubin platform has already adopted Spectrum-X Ethernet Photonics — the first CPO-based switch to enter mass production — which is the clearest evidence yet that co-packaged optics has crossed from proof-of-concept into actual commercial deployment. Nvidia’s own roadmap materials describe this as “CPO co-invention with ecosystem partners”: the first 1.6T silicon photonics chip built around new micro-ring modulators, the first 3D-stacked photonics engine built on TSMC’s process, high-power high-efficiency lasers, detachable fiber connectors for serviceability, and — the detail that matters most structurally — hundreds of patents licensed out to a roster spanning Coherent, Lumentum, Corning, Fabrinet, Foxconn, Sumitomo Electric, Senko, and TSMC, rather than held closed. That is Nvidia subsidizing and standardizing the supply chain underneath its own roadmap, the same playbook it has used with CUDA and NVLink — turning would-be competitors and suppliers into an ecosystem with a shared interest in its architecture winning.

The largest buyer of AI compute has told the market, in shipping silicon, capital commitments, and licensed IP, that the interconnect layer is no longer a procurement afterthought. It is now co-designed with the compute itself — and as optical connectivity extends further into the server, closer to the chip-to-memory link itself, it opens the door to disaggregating compute, memory, and networking entirely, and operating the data center less like a collection of fixed servers and more like a single, dynamically configurable system.

The Near-Term Bridge: VCSEL-Based Near-Package Optics

Co-packaged optics is the destination, but it is not yet the easiest way to ship volume today — CPO requires new packaging, new yield curves, and new supply chains to mature simultaneously. Broadcom’s answer to that gap is VCSEL-based Near-Package Optics (NPO): optical engines built on vertical-cavity surface-emitting lasers, mounted close to — but not integrated into — the switch package.

The appeal is pragmatic rather than visionary. VCSELs are a mature, high-volume, field-proven technology already manufactured at scale for 100G multi-mode applications, which means NPO can be deployed today on proven supply chains rather than waiting for CPO’s packaging ecosystem to mature. Broadcom’s 3.2T VCSEL-based NPO solution is positioned explicitly as a bridge: lower cost and lower power than legacy pluggables, small enough to preserve architectural flexibility, and good enough on wall-plug efficiency across a wide range of speeds that it doesn’t force operators to make an all-or-nothing bet on CPO timelines. For investors, NPO is worth tracking less as a long-term technology winner and more as a signal of sequencing — it is the architecture the industry is choosing to absorb demand while CPO’s manufacturing base catches up, and the companies supplying it (Broadcom prominently among them) get a multi-year revenue bridge while the bigger structural transition plays out underneath.

Two Complementary Paths Beyond Copper: Photonics and THz Radio-Over-Wire

It is worth grounding this in what a photonic chip actually is at the component level, since the term gets used loosely. A photonic integrated circuit replaces copper traces with optical waveguides etched directly into the chip: a laser generates light, an optical modulator encodes data onto it, ring resonators and couplers route and filter individual wavelengths, and a photodiode converts the signal back into an electrical one at the receiving end — all stages that copper interconnects simply don’t need, and all stages that have to be manufactured, packaged, and yielded at volume for any of this to ship at AI-cluster scale. That manufacturing complexity is precisely why the foundry and packaging layer (Tower, Fabrinet, TSMC) sits at the center of this entire build-out, not on its periphery.

Photonics is the approach getting the largest share of funding and standards attention today — Nvidia, Broadcom, TSMC, and the GF Fotonix coalition are all building toward it — and it is genuinely the right tool for the longest-reach, highest-bandwidth links, where the cost and complexity of converting electrons to light is worth paying. The more interesting story for 2026, though, is what is emerging alongside it.

AttoTude — a Menlo Park-based startup, and a Wing portfolio company — has built what we think is one of the more elegant architectural answers to the copper wall: THz radio-over-wire. Rather than converting electrical signals to light and back again, its AttoEngine drives terahertz-frequency radio signals directly through ordinary wire (AttoWire), borrowing the high-frequency design discipline of photonics while skipping the optical conversion stage entirely — along with the laser sources, modulators, and photodetectors that stage requires. It is a genuinely novel piece of systems engineering: getting radio signals to behave at terahertz frequencies over wire, at the reliability and reach AttoTude reports, is a hard problem that very few teams in the world are positioned to solve.

The result is a technology with real, differentiated advantages where it counts. The company reports per-lane speeds up to 448 Gb/s and reach up to 40 meters directly over wire — covering a substantial share of intra- and inter-rack distances — with fewer conversion stages translating directly into better reliability and lower power draw per bit than an optical link of comparable reach. That is not a marginal efficiency story; removing an entire conversion stage from the signal chain is the kind of structural advantage that compounds at scale. The company has raised $91 million to date across seed, Series A, and a $50 million Series B in 2026, and has begun a signal-analysis collaboration with Keysight — concrete signs of a credible, fast-moving, well-capitalized program with a genuine technical edge, not a paper architecture chasing a trend.

The practical picture for the AI data center of 2028–2030 is a layered fabric, not a winner-take-all contest: photonics carrying the longest-reach, highest-bandwidth links where its cost is justified, and AttoTude’s THz radio-over-wire taking on the shorter- and medium-reach links — a much larger share of total connections by sheer count — where a simpler, more reliable, more power-efficient signal chain is the deciding factor. Both approaches expand the addressable interconnect market rather than splitting a fixed one. We think AttoTude’s approach is one of the more capital-efficient and structurally advantaged ways to attack that expanding market, and it is the one we have backed.

On the Horizon: MicroLEDs — the Step After NPO

The sequencing logic that makes VCSEL/NPO a credible bridge today points to a follow-on question: what comes after CPO and NPO both reach volume? The most-cited answer at OFC and Computex 2026 was microLED-based optical interconnects — and it’s worth naming for exactly that reason, not as a standalone bet. Companies including Avicena, Kopin (in partnership with Fabric.AI), and MediaTek showcased microLED architectures, pitching them as a lower-power, higher-density alternative to laser-based optics for the shortest-reach, highest-count links inside the rack — precisely the segment where Tower’s and Fabrinet’s foundry constraints bite hardest, and where a non-laser-based approach could meaningfully ease the precision-assembly bottleneck described above. Fabric.AI has targeted a late-2026 demonstration of its “Neural I/O” platform, and at OFC 2026 every hyperscaler in attendance reportedly named I/O density as their central interconnect concern — with most specifically naming microLEDs as the candidate for that next step.

The honest framing: microLEDs are real enough to take seriously as a multi-year theme, and early enough that naming a winner today would be guessing. They belong on the watch-list — not as a fourth horse in this race, but as the clearest signal that the industry already assumes today’s CPO/NPO generation is a way-station, not an endpoint. That assumption, more than any single technology bet, is what should anchor a multi-year view of this space: the interconnect problem is durable enough that the industry is already funding its own replacement before the current generation has even shipped at volume.

Lumentum and Coherent: The Incumbent Optical Suppliers Re-Rating on AI

Lumentum and Coherent are the two large-cap, pure-play optical component companies riding the photonics side of this cycle most directly, and their recent results show the re-rating happening in real time.

Lumentum’s fiscal Q3 2026 revenue rose 90% year-over-year to $808.4 million, with non-GAAP gross margin and operating margin both hitting record highs of 47.9% and 32.2% — a margin profile that looks far more like a systems company than a commodity transceiver vendor. AI and cloud infrastructure now account for over 60% of Lumentum’s revenue, and management guides to roughly 85%+ year-over-year growth, anchored by two emerging product lines: optical circuit switches (backlog exceeding $400 million) and co-packaged optics, where it has booked a multi-hundred-million-dollar CPO order for delivery in the first half of calendar 2027.

Coherent posted Q3 FY26 revenue of $1.80 billion, up 27% year-over-year, with its Datacenter & Communications segment now generating roughly 75% of total revenue — effectively transforming what was once a diversified laser and materials company into an AI-datacenter-infrastructure supplier.

Both illustrate the same underlying shift: optical components have moved from a line item inside networking budgets to a primary determinant of how many GPUs a data center can actually utilize. The companies that can manufacture at the precision and volume AI clusters require are capturing share and margin simultaneously — and that combination, growth plus margin expansion, is the signature of a re-rating, not a cycle.

It would be incomplete, though, to frame this as a US-only story. Innolight (Eoptolink’s larger rival) is the global volume leader in 800G optical modules, with roughly 35% share of 2026 shipments, and posted FY2025 net profit up 108% year-over-year (and a further 262% in Q1 2026) — figures that rival or exceed the Western incumbents’ growth rates on a much larger unit base. Together, Innolight and Eoptolink supply an estimated 60% of Nvidia’s incremental 800G module orders, which means two Chinese manufacturers — not Lumentum or Coherent — currently control the single largest swing factor in how fast Nvidia can actually ship racks. Innolight’s 800G silicon photonics modules reportedly run yields above 92% at under 14W per module; Eoptolink’s linear pluggable optics cut power roughly 50% versus conventional designs and have already won orders from Meta and Amazon. The uncomfortable fact sitting underneath the Lumentum/Coherent re-rating story, then, is this: two Chinese manufacturers — not the Western component names investors gravitate toward — currently control the single largest swing factor in how fast Nvidia can actually ship racks, and the US re-shoring efforts described above are aimed at changing that over a multi-year horizon, not next quarter.

The Manufacturing Layer: Tower and Fabrinet Hold the Real Bottleneck

Behind the component brands sits a manufacturing and foundry layer that is arguably the more interesting story about who actually controls supply — because Tower Semiconductor and Fabrinet are, functionally, the bottleneck the entire interconnect re-rating runs through. Every optical architecture discussed in this piece — Nvidia’s CPO roadmap, Lumentum’s and Coherent’s component lines, Marvell’s and Broadcom’s switch programs — ultimately needs precision photonic and analog assembly at volumes that only a small number of foundries on earth can deliver. (AttoTude’s radio-over-wire approach is a structural exception worth noting here: by skipping the optical conversion stage entirely, it sidesteps this specific bottleneck — one more way its architecture diverges from, rather than competes directly with, the photonics supply chain.) Tower and Fabrinet sit directly across that chokepoint, which is what gives their backlogs the multi-year visibility and pricing power described below.

Tower Semiconductor has signed $1.3 billion in silicon photonics contracts for 2027 delivery from its largest customers, backed by $290 million in prepayments for capacity reservation — a financing structure that signals real scarcity, not speculative demand. Tower is guiding to a 2028 model of $2.8 billion in revenue and $750 million in net profit, levered substantially to its silicon photonics and silicon-germanium BiCMOS platforms.

Fabrinet, the contract manufacturer that assembles a large share of the industry’s optical transceivers, reported Q2 FY26 optical communications revenue up 29% year-over-year to roughly $833 million, with management projecting AI optics programs alone will exceed $150 million per quarter and that its new Chonburi Building 10 will push next-generation AI optics to over 50% of total revenue exposure.

These are capacity-constrained businesses in a capacity-constrained industry — precisely the setup that produces durable pricing power and multi-year revenue visibility once a backlog is booked. And it is a layer that benefits regardless of which architecture ultimately wins more share, since photonics, VCSEL/NPO, THz-over-wire, and microLEDs all require precision packaging at volumes only a handful of foundries can deliver.

The Architecture Race: Who Controls the Next Layer Up

The next layer up — system architecture, not component supply — is where the long-term winners will actually be decided, and 2026 produced the clearest signal yet of how seriously the largest semiconductor incumbents, switching vendors, and venture-backed challengers all take it.

Marvell laid out the most explicit thesis of the year. At Computex 2026, CEO Matt Murphy argued that AI infrastructure has moved through two prior bottlenecks — first compute (which built Nvidia), then memory — and that connectivity is next, because compute performance gains are now outpacing networking gains. He positioned Marvell as the only company able to address the entire connectivity stack of an AI data center, “from millimeter scale inside the package to kilometer scale between data centers,” and pointed to roughly $36 billion in cumulative M&A — including Inphi, Cavium, and now Celestial AI — as the basis for that claim.

It’s an assertive pitch — worth weighing against the fact that a CEO laying out a “we’re the only ones who can do this” thesis at a flagship keynote has every incentive to make the addressable market sound as large, and Marvell’s position in it as singular, as possible. The technical case behind it is more concrete: Marvell detailed the engineering stack behind its Coherent optical modules — fourth-generation silicon photonics, advanced CMOS DSP, and silicon-germanium broadband analog, integrated together rather than sourced piecemeal — which is the kind of vertically-integrated stack that’s genuinely hard to replicate, and the strongest evidence for the thesis independent of how Murphy chose to frame it.

That M&A thesis runs directly through Celestial AI, which Marvell has now brought in for a headline value of roughly $5.5 billion — though a meaningful share of that figure is performance-contingent earnout rather than committed capital. The acquisition buys Marvell the “Photonic Fabric” technology, built around an “Optical Multi-Chip Interconnect Bridge” that lets any point on one die connect optically to any point on another — the clearest evidence yet that the largest players see optical chip-to-chip interconnect as core infrastructure worth owning outright, not a component to be sourced.

Marvell also used Computex to ship hardware, not just thesis: the Teralynx T100, the industry’s first 102.4 Tb/s switch purpose-built for AI data centers, runs on a 3nm process, draws under 1,000W, and delivers roughly 25% lower power than competing solutions — available in conventional pluggable, co-packaged-copper, and co-packaged-optics configurations side by side, which is itself a tell about how unsettled the near-term architecture question still is even for the companies building it.

Around Marvell’s moves, the rest of the field is moving fast too. Lightmatter raised $155 million in a Series C and has positioned its “Passage” 3D photonic interposer — featuring dense wavelength-division multiplexing and co-packaged optics — to target supercomputers scaling to roughly 300,000 nodes. Ayar Labs, backed by both Nvidia and AMD, closed a $500 million Series E in March 2026 (total funding $870 million, valuation $3.75 billion) explicitly to scale volume production of optical chiplet interconnects — a clear signal that the largest compute vendors see independent optical I/O providers as strategically necessary rather than redundant to their in-house programs. Broadcom, pursuing both CPO and the nearer-term VCSEL/NPO bridge from the switch-ASIC side, and Marvell, now doubling down through Celestial AI and the GF Fotonix platform (alongside GlobalFoundries, Cisco, and Nvidia), are — per IDTechEx — locked in what is likely the defining strategic contest in the sector through the end of the decade.

Arista Networks is making the same bet from the switching side, with a pluggable-first lens. Rather than racing straight to co-packaged optics, Arista is pushing high-density liquid-cooled pluggable optics — its new XPO platform reaches 12.8 Tbps per module, a 4x front-panel density jump over current 1.6T OSFP designs (204.8 Tbps per OCP rack unit), while management says the architecture cuts AI networking racks by up to 75% and floor space by up to 44% versus conventional pluggable deployments. Arista’s framing matters because it is effectively a referendum on sequencing, not direction: the company is betting that liquid-cooled, ultra-dense pluggables can absorb several more product cycles of bandwidth growth before the industry is forced into the harder packaging and yield problems CPO still has to solve. That’s a more conservative read of the same transition Nvidia and Broadcom are pricing in by going straight at co-packaged designs — and for investors, Arista is the cleanest test of whether “good enough, shipping now” beats “structurally better, shipping later” in this specific cycle — and the fact that a switching incumbent of its scale is still betting heavily on pluggables is itself a data point that the CPO transition will be slower and more contested than the most aggressive roadmaps imply.

One more layer worth tracking beneath the architecture race: multi-wavelength lasers, which attack the bandwidth problem by putting more colors of light on a single fiber rather than more fibers in a bundle. Xscape Photonics raised $37 million in new funding in March 2026 (bringing total Series A funding to $81 million) and launched FalconX, an eight-wavelength laser device for AI data center networks, with a roadmap (ChromX) targeting 128+ wavelengths on a single chip — developed in partnership with Tower Semiconductor, reinforcing just how central that foundry layer is to every approach in this piece, not only CPO. Enlightra, a 25-person Swiss startup, raised $15 million to build chip-scale 8- and 16-channel multiwavelength lasers explicitly aimed at replacing copper in AI chip interconnects. The logic is straightforward and compounding: every additional wavelength multiplies the data a single fiber and a single set of transceivers can carry without adding fiber count, optical connectors, or rack space — which is exactly the kind of bandwidth-per-dollar and bandwidth-per-watt lever that becomes disproportionately valuable as I/O density, not raw laser count, becomes the binding constraint inside the rack. It is early-stage venture risk today, but the structural appeal is real: scaling bandwidth by adding wavelengths to existing fiber, rather than by adding fiber count or transceiver volume, is one of the few levers that can grow capacity without a proportional increase in the precision-assembly work the rest of this supply chain depends on — which is also why Tower, sitting at that assembly chokepoint, has chosen to co-develop the technology with Xscape rather than simply wait to manufacture it.

Implications for Investors and Builders

For investors, the read is this: the interconnect layer is shifting from a historically cyclical, commoditized, margin-poor sub-sector of optical networking into a genuine control point in the AI infrastructure stack — the same dynamic that turned HBM, advanced packaging, and power infrastructure into structural winners over the last three years.

The companies best positioned sit at genuine bottlenecks across multiple physical approaches at once: precision manufacturing capacity (Tower, Fabrinet), systems-level integration with the compute roadmap (Lumentum, Coherent, Broadcom, Marvell), and novel I/O architectures co-designed with the largest buyers — spanning photonic chiplets (Ayar Labs, Lightmatter, Celestial AI/Marvell) and the complementary THz radio-over-wire layer (AttoTude) — with VCSEL/NPO and microLEDs as two further approaches still establishing where, and how durably, they fit into that picture.

Two risks deserve explicit weighting. The first is timing: much of this revenue is backlog and 2027–2029 guidance, not booked trailing revenue — Marvell’s own Celestial AI numbers don’t ramp until late 2028, and a meaningful share of even the headline deal values (as with the “$5.5 billion” Celestial AI figure) is contingent, not committed. The second is concentration: a large share of demand traces back to a small number of hyperscaler and frontier-lab capex decisions that could shift abruptly, and a downturn in any one of them would ripple through the entire stack faster than diversified end markets would absorb it.

For founders and engineers, the message is more direct: interconnect bandwidth per watt, not raw FLOPs, is becoming the design constraint that determines which architectures are buildable at scale. Teams that treat the interconnect — whether optical, radio, VCSEL-based, or some hybrid — as a systems-design primitive co-designed from day one, rather than a procurement decision made late in the build, will hold a structural advantage in both cost and achievable cluster size over the next three to five years. The fact that Nvidia, Broadcom, and Marvell are each shipping multiple interconnect formats simultaneously is itself the clearest evidence that no one — including the companies with the most at stake — believes this question is settled yet.

Sources: Tom’s Hardware, Nvidia Newsroom and Technical Blog, Broadcom (company blog on VCSEL-based Near-Package Optics), IDTechEx, Digitimes, IEEE Spectrum, SemiEngineering, OFC 2026 and Computex 2026 industry coverage (Semtech, TSP Semiconductor, EE Times, ServeTheHome, SemiconAlpha, The Register, LEDinside, Converge Digest, Insight Media), SEC filings and investor releases (Lumentum, Coherent, Tower Semiconductor, Marvell, Arista Networks), Fabrinet investor commentary (Quiver Quantitative), Ayar Labs press materials (BusinessWire, DatacenterDynamics, The Register), Celestial AI and Marvell deal coverage (optics.org, Converge Digest, Optica/OPN, Marvell investor relations), Lightmatter coverage (TSP Semiconductor), Arista XPO and OFC 2026 coverage (arista.com, COMNEN), Xscape Photonics and Enlightra funding coverage (BusinessWire, HPCwire, The Quantum Insider, SiliconANGLE), AttoTude company site and press materials (attotude.com, BusinessWire, Keysight Newsroom), hyperscaler capex coverage (Goldman Sachs, Futurum Group, Fortune), and silicon photonics market sizing from Mordor Intelligence, MarketsandMarkets, Polaris Market Research, Precedence Research, GMI Insights, and Grand View Research. Disclosure: Wing is an investor in AttoTude.

We first started writing about hyperscaler ASIC programs and the Google TPU back in early 2024. The ASIC programs have expanded massively since. This post maps every layer of the TPU supply chain: silicon design, fabrication, packaging, memory, optical interconnects, and system assembly — with the key beneficiaries at each layer.

The popular narrative is that Google does everything themselves. The reality? Their infrastructure relies on a quiet, but critical, ecosystem of hardware partners. The architecture is Google’s. The implementation is not.

These design partners feed into a five-layer supply chain extending from chip architecture through to server rack assembly across six countries.

Architecture: What Ironwood Actually Is

Before mapping the supply chain, the architecture matters. Google’s seventh-generation TPU — Ironwood (TPUv7) — represents a fundamental design break, and its choices determine who wins supply chain share.

Each chip delivers 4.6 petaFLOPS of dense FP8 — ahead of Nvidia’s B200 at 4.5 PFLOPS — with 192 GB of HBM3E at 7.4 TB/s bandwidth via eight stacks. Manufactured on TSMC N3P with CoWoS advanced packaging, Ironwood is a chiplet design with two dies connected via a 6x-speed die-to-die link. The defining architectural bet is the network: 9,216-chip superpods interconnected via a 3D Torus topology and Google’s Apollo Optical Circuit Switch (OCS) — an all-optical fabric where a single switch draws ~100W versus ~3,000W for traditional electronic switching. At pod scale, that delta is structural.

2026 TPU shipment estimates span 3.3 million units on the supply-constrained end (DigiTimes Research, citing CoWoS packaging limits) to 4.6 million on the high end (Bank of America). The range reflects a genuine CoWoS allocation constraint, not demand uncertainty — Google’s order book is not the binding variable. ASIC server shipments overall are forecast to grow 64.2% in 2026 versus 43.8% for GPU servers, with Google accounting for the dominant share among hyperscalers.

2026 projected ASIC accelerator shipments by hyperscaler. Source: DigiTimes Research, TrendForce.

Silicon Design Partners

Google owns the TPU architecture and software stack. The implementation work is distributed across a growing set of ASIC design partners — all fabbing at TSMC — with deliberate competitive tension built in.

Broadcom — primary design partner

Broadcom translates Google’s architecture into a manufacturable ASIC layout, contributes high-speed SerDes IP, handles power delivery, and coordinates CoWoS packaging with TSMC. The structural position is the story: the contract runs through 2031, Broadcom carries a $73B AI backlog scheduled for delivery over the next 18 months, and the company holds >70% market share in custom AI accelerators globally. The financial trajectory follows from that position. Full-year FY2025 AI revenue reached $20B (+65% YoY); Q1 FY2026 came in at $8.4B (+106% YoY), with Q2 guided to $10.7B. JPMorgan forecasts $50B+ for full-year FY2026 — ~150% growth. CEO Hock Tan has stated “line of sight to $100B in AI chip revenue in 2027,” with Google and Anthropic representing the dominant share.

MediaTek — inference design partner

TPUv8 splits training and inference at the silicon level. Based on supply chain research (neither Google nor MediaTek has made a public announcement), MediaTek is designing the TPU 8i (”Zebrafish”) — a single compute die plus one I/O die plus six HBM3e stacks, targeting TSMC 2nm in late 2027. The cost profile is the thesis: Zebrafish targets 20–30% lower cost than the training variant. As inference grows to 75–80% of all AI compute cycles by 2030 (per industry estimates), a purpose-built, cost-tuned inference die has durable demand. For MediaTek, it is the entry ticket into hyperscale custom silicon — a market TrendForce sizes at $118B by 2033.

Marvell — in negotiation

Google is in active discussions with Marvell for two chips: a memory processing unit and an inference-optimized TPU variant. No contract is signed as of mid-2026. Marvell’s custom silicon business runs at a $1.5B annual rate across 18 cloud-provider design wins. Adding Google would be transformative. The multi-partner strategy — Broadcom for training, MediaTek for inference, Marvell for edge inference and memory processing — creates pricing competition and eliminates single-source capacity risk.

Fabrication and Packaging

TSMC manufactures every Google TPU. Ironwood runs on N3P; TPUv8 targets N2, with tape-outs planned for 2026–2027. TSMC also handles CoWoS advanced packaging — the step integrating compute chiplets with HBM3e stacks. TSMC is targeting 140,000 CoWoS wafers per month by end-2026, a 4x expansion from early 2024. Even at that scale, CoWoS remains an allocation market — which is why supply chain researchers and Bank of America disagree on 2026 TPU volumes by more than a million units.

CoWoS packaging requires ultra-high-density ABF substrates. Ibiden and Shinko Electric dominate this layer. AI server substrates carry approximately 3x the pricing premium of standard server substrates due to layer count and routing complexity. New fab construction takes 2–3 years and hundreds of millions in capital — supply cannot respond quickly to demand.

High-Bandwidth Memory

Ironwood’s 192 GB per chip, 8-stack HBM3E configuration makes HBM arguably the most consequential component constraint in the program. The chiplet design — two compute dies connected via a die-to-die interface rather than one monolithic die — is partly dictated by TSMC’s reticle size limit at N3P, but it also maximizes the number of HBM stacks each die can directly service: four stacks per die, each connected via a dedicated 512-bit memory bus. That physical arrangement is why HBM supply, not compute logic, is the effective ceiling on Ironwood production volumes.

Samsung holds approximately 60% of HBM3E supply to Google via Broadcom for Ironwood, with HBM4 already qualifying through Broadcom’s test process for the next generation. SK Hynix supplies the remaining ~40%. Micron is not a meaningful factor in this supply chain as of mid-2026.

Optical Interconnects — a differentiated layer

This is the layer most investors underestimate. Google’s Apollo OCS architecture is a full architectural break from conventional switching, and the qualified supplier base is narrow.

Lumentum — Apollo OCS system

Lumentum supplies the MEMS-based optical circuit switches at the heart of Apollo. These switches use micro-mirrors to enable direct fiber-to-fiber connections, eliminating optical-electrical-optical conversions. A single Apollo switch draws ~100W versus ~3,000W for a comparable electronic switch — a 95% power reduction. Lumentum’s primary commercial product is a 300×300 OCS at approximately $150,000 per unit (~$500/port). In 2025, Google deployed an estimated 12,000–15,000 units. Lumentum is the sole-source supplier for this component; there is no qualified alternative in production.

Innolight and Eoptolink — 800G+ transceivers

Every cross-cabinet link in an Ironwood pod runs on 800G or 1.6T optical transceivers. TrendForce projects Innolight — the larger of the two and Google’s primary silicon photonics collaborator on 1.6T development — will capture ~80% of Google’s 800G+ optical module orders in 2026. Eoptolink captures most of the remainder; it is a second-tier supplier by volume but strategically important for supply chain resilience. The company posted 179% revenue growth in 2024, driven by cloud operator direct sales, and has built a manufacturing facility in Thailand specifically to serve North American demand outside Chinese export-control exposure. Collectively, Google’s Ironwood architecture is projected to push 800G+ transceiver share above 60% of all optical module demand by 2026 (TrendForce).

System Assembly — Server ODMs

TPU chips and optical components must be integrated into server platforms and racks. Taiwan-based ODMs dominate this work. Inventec is the most clearly confirmed primary ODM for Google’s TPU server builds, with management publicly expressing confidence in double-digit ASIC server shipment growth in 2026. Wiwynn (a Wistron subsidiary) is actively expanding SMT production capacity in Malaysia for ASIC server assembly. Quanta, Foxconn, and Mitac are all competing for additional allocation.

ASIC server shipments are forecast to grow 64.2% in 2026 versus 43.8% for GPU servers. The margin dynamic favors ODMs: ASIC servers carry lower unit prices than GPU platforms but require more customized design work and longer production runs, which compresses per-unit revenue while improving margin mix. The net effect is that ODM revenue from ASIC servers grows more slowly than volume, but profitability improves as ASIC share of the revenue mix increases.

Beneficiary Summary

Closing Thesis

The companies with clearest structural exposure are Broadcom (silicon implementation, locked through 2031), Samsung (HBM memory, primary qualified supplier), and Lumentum (optical switching, sole-source). These are not cyclical trades on AI capex — they are design partners with multi-year roadmaps and switching costs measured in years.

At $175–185B in committed 2026 capex and 3.3–4.6M projected TPU shipments, the question is no longer whether Google’s custom silicon program is real. The question worth stress-testing is what breaks the thesis. Three scenarios are worth watching: first, CoWoS yield issues at TSMC N2 that compress 2027 volumes below the 5M unit consensus — this directly impairs every vendor in the stack. Second, Samsung HBM4 qualification delays that concentrate more supply at SK Hynix and introduce a single-source risk Google has spent years trying to avoid. Third, Nvidia closing the inference cost-per-token gap faster than expected through Blackwell derivatives — this doesn’t threaten Broadcom’s training revenue, but it limits how aggressively Google can price TPU inference access externally, which is increasingly the commercial justification for the capex.

None of these scenarios break the supply chain. They delay it and compress near-term returns for the vendors most dependent on volume ramp. The structural position of the Tier 1 beneficiaries — design lock-in, sole-source components, multi-year agreements — survives all three. What changes is the timeline, not the destination.

Sources / Recommended readings:
SemiAnalysis: Google TPUv7 — The 900lb Gorilla
Google Cloud: Inside the Ironwood TPU codesigned AI stack
The Next Web: Google assembles four-partner chip supply chain
TrendForce: Global AI Server Shipments Forecast 2026
TrendForce: Google's 800G+ Optical Transceiver Push
KED Global: Samsung leads HBM for Google Ironwood (~60% share)
DigiTimes: Google TPU stays strong, Inventec confident 2026
DigiTimes: Google TPU demand vs CoWoS and memory constraints
Gazettabyte: Lumentum OCS for AI data centres
Oplexa: Broadcom Google TPU Deal 2026
CNBC: Alphabet plans $80B equity raise
Broadcom: FY2025 Annual Results
Tom's Hardware: Custom AI ASICs — state of play, May 2026

TL;DR — Five things to know

1. AI racks go from 120 kW today to 600 kW by 2027. Power infrastructure value per rack jumps from $36K to $398K+ — more than 10× baseline. This is not incremental.

2. The architecture shifts completely — AC→50V DC becomes AC→800V DC (HVDC). Silicon Carbide and GaN are the enabling materials. NVIDIA has already picked its semiconductor partners.

3. Air cooling is effectively dead above 40 kW. The liquid cooling market grows from $5.5B to $25.8B by 2035. Every new hyperscale campus is being designed liquid-native from the ground up.

4. Power transformers now carry 4-year lead times. More than 50% of U.S. data centers planned for 2026 face delays. This is a structural bottleneck, not a supply chain hiccup.

5. The investment thesis spans five layers: grid infrastructure → facility power → rack PSU → power semiconductors → chip-level VRM. Each has different risk, margin, and timing.

Five Generations. Five Times the Power. Twenty-Four Months.

A GB200 Blackwell rack today draws 120 kilowatts. That’s already forcing hyperscale operators to redesign power distribution floors and pull in supplemental cooling hardware mid-build. GB300 takes that to 140 kW. Vera Rubin, shipping in 2026, clears 200 kW. Vera Rubin CPX hits 380 kW. And Vera Rubin Ultra — paired with NVIDIA’s new Kyber rack architecture — lands at 600 kW per rack.

Six hundred kilowatts. In a single rack. Roughly the power draw of 600 American homes, packed into a steel enclosure seven feet tall and two feet wide. A 1,000-rack AI cluster at Vera Rubin Ultra density requires 600 megawatts of utility-grade power — the average electrical load of San Francisco.

The economic consequence, per Morgan Stanley Research, is equally direct. Power infrastructure value per AI server rack jumps from $36,000 with GB200 to $398,160 with Vera Rubin CPX — a 1,005% increase — and exceeds 10× baseline with Vera Rubin Ultra. The power electronics industry isn’t a side story in the AI buildout. It’s one of the central economic narratives.

Source: Morgan Stanley Research, Exhibit 7

The Architecture Has to Change. Completely.

Today’s data centers run on a power architecture that was never designed for what’s being asked of it. AC from the grid converts to low-voltage DC inside each server’s PSU. Fine at 10 kW per rack — a number that grew only incrementally for decades. Now it’s growing exponentially, and at 600 kW the physics turn against you hard.

Source: AEIS

The problem is Ohm’s law. Power loss in a conductor equals current squared times resistance (P = I²R). Low-voltage DC forces you to move the same wattage at much higher current, and losses scale with the square of that current. The copper in your busways heats up. The conversion stages multiply. A system losing 8–10% of its power at 120 kW is wasting 48–60 kW per rack at 600 kW — waste heat equivalent to the HVAC load of a small office building, generated inside a 7-foot cabinet.

NVIDIA’s answer, baked into Vera Rubin CPX and Ultra, is a complete switch to High-Voltage Direct Current at 800V. A centralized Silicon Carbide rectifier converts utility AC directly to 800V DC. High-voltage distribution across the rack cuts resistive losses sharply. Fewer conversion stages. Less heat. More compute per watt delivered.

This is a platform transition, not an upgrade cycle — comparable in kind to spinning disk to NAND, or mechanical relays to solid-state switching. Every piece of rack-level power infrastructure gets redesigned. Every PSU, busbar, connector, and voltage regulator module needs re-qualification for 800V operation. That qualification cycle runs 12–24 months per generation. Vendors who clear it first hold a durable position in that GPU lifecycle. Vendors who don’t are excluded from the design entirely.

Vertiv has a full 800V HVDC product line slated for 2026. ON Semiconductor and Navitas Semiconductor have both disclosed formal NVIDIA collaboration agreements for the 800V transition — the kind of design-in that locks a vendor into a GPU generation’s revenue cycle for 3–5 years.

Silicon Carbide and GaN: The Materials That Make It Possible

Silicon has a hard ceiling around 650V. It cannot efficiently switch at the voltages and speeds 800V HVDC demands. The answer is wide bandgap semiconductors — materials with a larger energy gap between valence and conduction bands, which translates directly to higher breakdown voltage, higher switching frequency, and lower on-resistance.

Two materials carry the load. Silicon Carbide handles the high-voltage, high-current front end of HVDC conversion. A 1200V SiC MOSFET replacing a legacy silicon device delivers a 25–40% reduction in conversion losses at the rectification stage. Gallium Nitride runs the other side — operating at extremely high switching frequencies in the DC-DC stages that deliver final voltage to the GPU die, enabling smaller magnetics and better transient response in a fraction of the footprint.

NVIDIA’s vendor selections are the clearest market signal available. Navitas Semiconductor’s GaNFast and GeneSiC technologies are in the reference design for Vera Rubin generation power systems. ON Semiconductor’s SiC MOSFET collaboration targets the 800V rectification stage specifically. Infineon — the world’s largest power semiconductor company at roughly €16 billion in annual revenue — has 1200V SiC devices in active qualification for AI data center applications, with manufacturing scale no smaller peer can replicate at hyperscale volumes.

The substrate bottleneck. Most SiC device manufacturers — ON Semi, STMicro, Infineon — depend on SiC wafer supply from a tight group of substrate producers. Wolfspeed was the dominant U.S. source. That thesis now requires careful handling: Wolfspeed filed for Chapter 11 in early 2025, crushed by its aggressive capacity expansion program colliding with softening EV demand. The substrate leverage position is still structurally real — the wafers are still needed, the manufacturing assets didn’t vanish — but the equity is a restructuring story, not a clean infrastructure compounder. Meanwhile, Chinese manufacturers (SICC, TankeBlue, Sanan IC) are expanding capacity aggressively, applying downward pressure on substrate pricing that will ultimately benefit device makers and end customers while compressing whoever inherits Wolfspeed’s supply position post-reorganization.

Air Cooling Is Already Dead. The Industry Just Hasn’t Filed the Death Certificate.

Conventional air cooling has a practical density ceiling around 30–40 kW per rack. Above that, the volume of air you’d need to move exceeds what you can force through a rack enclosure at reasonable static pressure. Fans consume more power. Hot spots form. The economics invert — you spend more electricity removing heat than you would have spent throttling the compute.

At 120 kW, operators are already deploying rear-door heat exchangers and direct-to-chip liquid loops as a patch. Vera Rubin at 200 kW makes direct-to-chip mandatory. CPX at 380 kW pushes into immersion cooling territory. Vera Rubin Ultra at 600 kW strongly favors two-phase immersion — liquid refrigerant that boils on contact with hot components, with vapor recondensed and recirculated — though high-flow single-phase direct liquid cooling remains physically viable depending on the specific thermal design. This is industrial cooling technology borrowed from semiconductor fabs and chemical plants, now being deployed at internet infrastructure scale.

The capital intensity here is severe, and not in a “spend more to upgrade” sense — in a “start over” sense. A facility built for 40 kW air-cooled racks cannot be incrementally retrofitted to handle 200+ kW liquid-cooled systems. Floor loading, piping, leak detection, CDU placement, and electrical distribution all have to be redesigned from scratch. Greenfield hyperscale campuses announced today are going in liquid-native. Legacy colo providers face a binary: spend heavily to rebuild, or cede the AI workload tier entirely to the hyperscalers who built for this from day one.

The market trajectory reflects this. Liquid cooling at roughly $5.5 billion in 2025 is projected to reach $25.8 billion by 2035 — approximately 18–20% CAGR, driven almost entirely by AI infrastructure density requirements. Cloud providers dominate the current installed base, but enterprise AI deployments will extend that demand well beyond the hyperscale tier.

Transformers: The Four-Year Bottleneck Nobody Talks About Enough

Every kilowatt of AI compute connects to the utility grid through a transformer. Not metaphorically. A physical transformer — a custom-engineered device wound with high-purity copper and grain-oriented electrical steel, filled with insulating oil, weighing anywhere from 5 to 500 tons — sits at the boundary of every data center and the grid that feeds it. There is no shortcut around this device.

High-capacity units now quote 4-year lead times. Standard power transformers are at 128 weeks. Generator step-up units (GSUs) are at 144 weeks. Prices for power transformers have risen 77% since 2019; GSUs are up 45%. GSU demand itself increased 274% between 2019 and 2025. More than 50% of U.S. data centers planned for 2026 are expected to be delayed or canceled specifically because specialized electrical equipment cannot be sourced in time.

The causes are structural. U.S. transformer manufacturing capacity hasn’t meaningfully expanded in 15 years. The critical raw material inputs — grain-oriented electrical steel (GOES) and high-purity copper — are sourced primarily from Japan, South Korea, and increasingly China. Building a new transformer factory takes 3–5 years from capital commitment to first unit out the door. Nobody is going to solve this in a budget cycle.

Substations compound the problem. A large AI campus needing 100–500 MW of utility interconnect requires switchgear, protection relays, metering equipment, and civil infrastructure — each with its own lead time stack. U.S. grid interconnect permitting averages 3–5 years. The interconnect queue at regional transmission organizations currently holds over 2,600 GW of pending capacity requests. Total U.S. installed generation capacity is approximately 1,200 GW. The queue is more than double the grid.

The AI Power Stack: Five Layers from Grid to GPU

Power doesn’t teleport from the utility grid to an NVIDIA GPU. It moves through five distinct layers of infrastructure, each with its own supply chain, lead times, and companies controlling the chokepoints.

Layer 1 — Grid & Substation Infrastructure High-voltage transformers, switchgear, substations. The bottleneck before the building even exists. Lead times now 2–4 years. More than 50% of U.S. data centers planned for 2026 face delays here.

Layer 2 — Data Center Power Distribution & UPS PDUs, UPS systems, busways, coolant distribution. Everything between the utility interconnect and the server rack. This is where facility-level critical power lives.

Layer 3 — Rack-Level Power & PSU Server power supply units and power shelves. Currently converting AC to 50V DC per server. The 800V HVDC transition — arriving with Vera Rubin CPX in 2027 — redesigns this entire layer from scratch.

Layer 4 — Power Semiconductors (SiC & GaN) The materials that make the 800V transition physically possible. Silicon Carbide handles high-voltage rectification; Gallium Nitride handles high-frequency low-voltage conversion. Together they deliver 25–40% fewer conversion losses versus legacy silicon.

Layer 5 — Point-of-Load & Chip-Level VRM Voltage regulators and power management ICs that deliver precise, tightly regulated voltage directly to the GPU die. Highest dollar content per AI server. Smallest physical footprint. Closest to the compute.

Each layer has different lead times, different competitive dynamics, and different exposure to the 800V transition. The companies controlling the chokepoints at each level are covered below — but the macro argument stands on its own.

The Physics Don’t Negotiate

Everything else in AI infrastructure is debatable: which model architecture wins long-term, which cloud provider takes share, which REIT lands the best interconnect deal. The power problem is not debatable. It is a direct consequence of semiconductor physics and thermodynamics — more compute per chip requires more power per chip, more chips per rack requires more power per rack, and neither of those relationships bends for anyone’s financial model.

The scale of the requirement — 600 kW per rack, 4-year transformer lead times, an HVDC transition touching every electrical component from the substation to the GPU — is measurable, predictable, and already locked into NVIDIA’s product roadmap. These are not projections. They are engineering specifications.

The companies in this stack are not speculating on AI demand. They are selling into a confirmed equipment cycle with 2–4 year backlog visibility and architectural transitions that reset design-win competition every 18–24 months. For investors willing to work through the layers, the power electronics theme may be one of the most grounded, data-supported ways to own the AI infrastructure buildout — without paying for hope.

Note: If you are a founder/operator building in this space, please reach out to me.

Three companies — SK Hynix, Samsung, and Micron — own that bottleneck. They control approximately 96% of DRAM and effectively 100% of High Bandwidth Memory (HBM). HBM is now the single most economically critical component inside Nvidia’s accelerators, contributing >50% of the bill of materials of a Blackwell Ultra (B300) and rising further on Rubin. Historically a cyclical industry, memory suppliers are seeing above prior peak-cycle gross margins today in excess of 70%.

At peak-cycle HBM economics, SK Hynix is beginning to outearn the hyperscaler businesses like Meta that buy their products. Combined, the memory triopoly is projected to generate 2–3x Meta’s net income in 2026.

Compute Scales Faster Than Memory

Every Nvidia generation has run into the same wall: memory. Memory capacity per GPU grew ~18x from P100 to Rubin Ultra in 11 years. FP compute grew approximately 500x over the same period.

The implication is structural: every successive AI generation becomes more bandwidth-constrained. Memory has gone from ~20% of GPU BOM (A100) to ~55% (B200). For Rubin, HBM may exceed the cost of the logic die itself. We are beginning to approach physics limits with how much HBM can be placed on the processor. GPU shoreline is fixed by die perimeter, and HBM stacks can only sit along two of the four edges — which is why Rubin Ultra moves to 16 stacks by going taller, not wider.

HBM Bill of Materials and Margin Capture

A B200 has 8 HBM3E stacks at 24 GB each (192 GB total). HBM share of GPU BOM is ~52% on B200, ~58% on B300 (Blackwell Ultra, 288 GB HBM3E 12-Hi), and tracking ~62% on Rubin (HBM4) based on early ASP guides.

Nvidia sells B200 at $30,000–40,000 list. Of every incremental B200 dollar, roughly $0.10–0.15 flows to HBM suppliers as gross margin contribution. SK Hynix captures ~45–50% of that. Samsung ~30%. Micron ~22%.

Memory suppliers are now tied to Nvidia’s revenue trajectory more tightly than perhaps any supplier-customer relationship in modern semiconductor history.

Cost per Gigabyte: The Brutal Hierarchy

Memory is not one market. It is a stratified stack with six orders of magnitude in cost difference.

HBM3E is ~6x DDR5 per gigabyte but ~25x the bandwidth per stack. On a $/(GB·bandwidth) basis, HBM is the only economically rational choice for AI workloads above a certain compute density.

SRAM is two to three orders of magnitude more expensive than HBM per gigabyte. And SRAM scaling has stalled. From TSMC N7 to N5, SRAM density improved 5%. From N5 to N3, less than 5% again. The historical assumption that on-die cache would keep up with logic has failed.

This is the SRAM scaling cliff. Every AI architecture that depends on large on-die caches (Cerebras, Groq-style approaches) faces an increasingly hostile cost curve. HBM remains the only memory technology that combines large capacity, high bandwidth, and acceptable cost.

The Memory Wall, Quantified

AI’s core architectural problem is no longer compute scarcity. It is memory starvation.

AI workloads are bound by three physical constraints, not by FLOPS:

1. Modern GPUs require far more compute per byte than transformer workloads naturally generate. Every GPU has a balance point — how many math operations it can do for each byte of memory it can fetch. For V100 (2017) the balance point was ~125 FLOPS per byte. For B200 today it is ~560 FLOPS per byte. To run the GPU flat-out, software needs to do that many operations on each byte it touches. Transformer decode workloads fall dramatically short of that threshold. Typical decode workloads only do 1–10 operations per byte fetched — they spend most of their time waiting on memory. The result: AI silicon often runs at 15–35% of peak FLOPS during inference. Every additional TB/s of HBM bandwidth converts directly into utilization recovery and lower cost-per-token.

2. KV cache now dominates inference memory. During inference, every prior token’s “key” and “value” representations have to stay in memory for the next token to attend to them. This is the KV cache, and it grows with context length × batch size. Llama 3 70B at 128K context with batch size 32 needs ~80 GB just for KV cache — more than the model weights themselves. At 1M context, KV cache exceeds 600 GB per request. This is why H200’s 141 GB and B200’s 192 GB are a step change versus H100’s 80 GB: not because the weights got bigger, but because the cache did.

3. Moving data costs more energy than doing math on it. Moving one bit across HBM costs about 7 picojoules. Across off-package DDR5: 80 pJ. From on-die SRAM: 0.1 pJ. Doing an 8-bit multiply on a 4 nm node: 0.05 pJ. At trillion-operation inference scales, data-movement energy exceeds compute energy by 3–10x.

   This is why every architectural response — HBM4 with logic base dies, stacked SRAM, processing-in-memory — is fundamentally about reducing joules-per-byte moved.

The economic translation: HBM bandwidth is the highest-leverage capex decision in AI infrastructure. For modern transformer workloads, the marginal value of additional HBM bandwidth often exceeds its marginal cost.

What an HBM Stack Actually Is

Source: High-Bandwidth Memory, Semi Engineering

Before getting into vendor differences, a quick engineering picture. HBM isn’t a chip — it’s a small skyscraper of chips.

An HBM3E 12-Hi stack is twelve DRAM dies sitting on top of a logic base die. Each DRAM die is ground down to about 30 μm thick — roughly a third of a human hair — so the whole tower fits inside the package. The dies are connected vertically by through-silicon vias (TSVs): copper-filled holes drilled through each die, 5–10 μm wide and spaced ~50 μm apart. A 12-Hi stack has 1,200–1,800 TSVs per layer for data and power, plus thousands more for signal integrity.

The stack sits next to the GPU compute die on a silicon interposer. The interposer is the wiring layer that carries the 1,024-bit (HBM3) or 2,048-bit (HBM4) bus between memory and compute at trace densities and impedances that organic substrates can’t match. For Nvidia accelerators, the relevant implementation is TSMC’s CoWoS-L.

Three things make HBM hard to build:

• Heat. Each stack dissipates 4–8 watts. That heat has to escape upward through twelve thin silicon layers and the adhesive between them. The thermal limit is why H100 launched in 2022 with only five of six possible HBM3 stacks active — the sixth bay was reserved as thermal and yield margin.

• Yield compounds. A single bad TSV kills the entire stack. At 99% per-die yield, a 12-Hi stack yields 88.6%. At 95% per-die, only 54%. This is why per-die yield matters more than headline density at HBM scale — and why SK Hynix’s 75–80% 1bnm yield is a wider competitive moat than the number alone suggests.

• Wider IO beats faster IO. HBM4 doubles the interface from 1,024 to 2,048 bits but keeps the per-pin speed roughly the same as HBM3 (~6.4 Gbps). Wider-and-slower uses less power per bit and yields better than narrower-and-faster at modern interposer densities. This single choice forces the base die onto a logic foundry node (TSMC N5/N3) instead of a DRAM line and determines which vendors can build it.

There are two ways to physically stack the dies:

• MR-MUF (Mass Reflow with Molded Underfill) — SK Hynix’s proprietary method. Bond and fill the gaps between dies in a single thermal cycle. Better heat conduction, fewer voids between layers.

• NCF (Non-Conductive Film) — Samsung’s method. A pre-applied film is bonded layer-by-layer. Cheaper equipment, slower per stack, lower yield historically. Samsung’s revised “Advanced TC-NCF” closed but did not eliminate the gap for HBM4.

Layer count keeps climbing. HBM4 supports 16-Hi configurations — double what HBM3 launched at. The earlier 720 μm packaging ceiling could not accommodate 16-Hi stacks, forcing the standard to expand to 775 μm in late 2024. The taller envelope means microbumps remain viable for HBM4, and the long-anticipated transition to direct copper-to-copper bonding has slipped to HBM4E or later.

Once copper-to-copper bonding arrives, interconnect pitch shrinks dramatically while removing the bump gap between dies entirely — enabling higher stack density and lower power per bit. The tradeoff is a new bottleneck: hybrid bonding tool capacity.

HBM market share, 2024–2026E. The SK Hynix monopoly era is over; Samsung's HBM3E 12-Hi qualification and Micron's ramp have compressed the leader's share by 7 points. Total HBM revenue still compounded >100% in 2025.

The Triopoly in Numbers

DRAM is a triopoly. HBM is no longer a SK Hynix monopoly — Samsung’s HBM3E 12-Hi qualification at Nvidia in Q4 2025 and Micron’s ramp have compressed the leader’s share by ~7 points in 18 months. But total HBM revenue continues to compound at >70% YoY, so all three are growing absolutely.

Through five DRAM cycles since 2010, the industry consolidated from 10+ vendors to three. Each downturn pushed marginal capacity out. The result is the highest sustained pricing power in the history of memory: HBM3E ASPs are 6–8x conventional DDR5 per gigabyte, HBM4 ASPs are ~10x, and capacity remains fully sold out through 2027.

SK Hynix — The HBM Leader

The #1 HBM supplier at ~45% share, sole supplier of HBM3 to Nvidia’s H100, and now the lead supplier of HBM4 into Rubin.

The technical moat is two-fold. SK Hynix’s 1bnm DRAM yield runs 75–80%, versus Samsung’s 60–65% on 1cnm — a gap that compounds across 12 stacked dies. Its proprietary MR-MUF stacking process keeps it alone above 80% yield on HBM3E 12-Hi.

Looking forward: SK Hynix shipped first HBM4 12-Hi samples to Nvidia in March 2026 — a 6–9 month lead over the competition into the Rubin socket. The HBM4 base die is fabricated at TSMC N5, coupling the memory and logic moats at the most advanced node.

Main risk: Nvidia is ~42% of total revenue, the tightest single-customer relationship in modern semiconductors.

Samsung — The Vertically Integrated Comeback

The largest memory company by capex, but it lost the HBM3 cycle. Thermal failures at Nvidia in 2023, yield misses at the 1anm node, and slower TC-NCF stacking left Samsung well behind.

The story flipped in 2026. 1cnm DRAM yields converged to ~65% in Q1 2026. HBM3E 12-Hi qualified at Nvidia in November 2025. “Advanced TC-NCF” closed the stacking gap. And AMD’s MI400 is now committed primarily to Samsung HBM4 — the first major AI socket Samsung has fully won this cycle.

The structural advantage no one else has: Samsung is the only memory company that also owns its own logic foundry. As HBM4E moves to customer-specific logic base dies in 2027, Samsung can co-design memory and base die without crossing a corporate boundary.

Main risk: Still 6+ months behind SK Hynix on HBM4. The HBM4E generation in 2027 is where Samsung’s integration thesis actually has to prove out.

Micron — The American Hedge

The smallest of the three, but with two distinct advantages.

A different DRAM cell architecture. Micron uses an “open bit-line” topology with vertical capacitors where SK Hynix and Samsung use planar designs. Micron claims ~30% lower active power on HBM3E at the same bandwidth (vendor-published, directionally credible).

US production. Micron is the only material DRAM manufacturer with US fabs. Idaho ID-1 opened October 2025; the New York Clay megafab was pulled forward to 2027. A $6.1B CHIPS Act grant and $7.5B in loans underwrite a multi-decade capex floor that no Asian competitor has.

Looking forward: HBM3E 12-Hi shipping into B300 since Q4 2025. HBM4 sampling at Nvidia in Q2 2026, volume H1 2027 — roughly parity with Samsung, ~6 months behind SK Hynix. HBM has gone from <2% of Micron revenue two years ago to ~26% today — a faster mix transition than either competitor went through.

Why Not More Players?

Greenfield DRAM fab cost in 2024: $15–25B. Lead time: 4–5 years. Yield ramp: 18–36 months. HBM-specific packaging capex (TSV, CoWoS-equivalent): another $2–5B per node.

Total cumulative capex required to enter the HBM market at material scale (10% share): ~$30–50B. Time required: 7–10 years.

No new entrant is plausible. Chinese efforts (CXMT, YMTC) are 3–5 nodes behind and structurally blocked from EUV. Indian, Japanese, and Taiwanese efforts are early-stage. The triopoly is durable for at least a decade.

Structural Observations

Not investment recommendations. Five things to track across the cycle:

1. SK Hynix’s HBM4 lead is the highest-margin product in memory history. Watch Nvidia concentration (42% of revenue) and hyperscaler dual-sourcing — visible already on AMD MI400 (Samsung) and Trainium 3 (Micron + Samsung).

2. Samsung’s recovery is data-supported, not speculative. 1cnm yield convergence, HBM3E 12-Hi qualification, and the MI400 win ended the “structurally behind” thesis. Open question: does foundry-plus-memory integration matter at HBM4E?

3. Micron has crossed from optionality to fundamentals. Fully booked HVM through 2026, 24% share, CHIPS Act margin floor. Main risk is HBM4 schedule slip.

4. TSMC is the silent fourth participant. CoWoS-L sold out through 2027 despite ~3x capacity expansion. Memory and logic moats are now coupled at the most advanced node.

5. Equipment vendors capture proportional margin. HBM capex intensity per wafer is 3–5x prior DRAM cycles — flowing to lithography, TSV etch, deposition, and hybrid-bonding tools.

Where Startups Are Trying to Break In

A wave of startups is either trying to route around HBM or build new abstractions on top of it. Three categories matter.

1. Memory Processing Units (MPUs). Compute placed inside the memory array itself, so data never has to cross the package. The architectural bet: as data movement comes to dominate energy and latency, the chip’s center of gravity shifts toward memory with compute as the satellite. Several early-stage startups have tested working silicon on this concept, but the category remains pre-commercial — useful as a forward indicator of where the industry may head, not yet a commercial threat to HBM.

2. Custom HBM-plus-SRAM accelerators. Transformer-specific ASICs that combine HBM for capacity with oversized on-die SRAM for bandwidth. Cerebras WSE-3 carries 44 GB of on-die SRAM at 21 PB/s aggregate — no HBM at all. Groq’s LPU goes further with ~230 MB SRAM per chip and statically schedules weights across hundreds of chips. MatX pairs HBM3E/HBM4 with large on-die cache, betting that workload specialization can beat Nvidia’s general-purpose silicon on tokens-per-dollar.

3. Memory fabrics. Connective tissue around memory rather than memory itself. Astera Labs (CXL switches and memory expanders, now public) leads the commercial wave. Eliyan’s NuLink offers organic-substrate chip-to-chip interconnect competitive with silicon interposers — potentially loosening TSMC’s CoWoS chokepoint. Enfabrica is building a converged network-and-memory fabric that pools HBM and DDR across a rack. Celestial AI is pushing photonic interconnects with claims of 10–100x lower energy per bit between memory and compute.

None of these displaces HBM as the bottleneck. Each either accepts HBM as the foundation and extracts more performance around it, or routes around HBM in workloads narrow enough for SRAM to substitute. That is exactly why the triopoly remains durable.

Summary

The memory triopoly is the most concentrated, most capital-intensive, most expensive chokepoint in the AI server. HBM is now 50%+ of accelerator BOM. Memory bandwidth grows ~2x per Nvidia generation while compute grows 5–10x. The gap widens. The triopoly captures the margin.

Nearly every architectural response — HBM4 logic base dies, stacked SRAM, PIM, MPUs, CXL fabrics — ultimately reinforces the same bottleneck. The next decade of AI capex driven by Nvidia and TSMC flows into the memory triopoly more than into any other layer of the stack.

The bottleneck is the moat.

That isn’t a 2x or 3x problem — it is an order-of-magnitude shift in machine-hours per shipped unit. Multiply across the millions of accelerators shipping into datacenters over 2025–2027 and you get a structural capacity squeeze in semiconductor test, with two companies — Advantest and Teradyne — sitting on the chokepoint.

At a glance

• Test time per die: ~50s mobile SoC → 20+ minutes for AI accelerator

• CY25 ATE TAM: $9.0B, +50% YoY

• Advantest SoC tester share: 56% → 66% in twelve months

• Teradyne Q1’26 revenue: +87% YoY, ~70% AI-driven

• Combined duopoly share: ~90%+

Why test time exploded

Four physics-driven dynamics are simultaneously inflating test surface area, test pattern count, and required test conditions. Each adds machine time linearly. Together they compound.

1. Die size at the reticle limit. Blackwell B200 packages two ~814mm² dies — each at the EUV reticle limit — connected via a 10TB/s NV-HBI interconnect. Test pattern count scales with transistor count (B200: ~208B transistors) and IO complexity. A reticle-limited die requires 2–3× the test patterns of a 7nm-era SoC.

2. Chiplet integration. UCIe and proprietary D2D fabrics let designers compose accelerators from 4–8 chiplets. Every chiplet must be screened as a Known-Good-Die (KGD) before assembly because a single defective die scraps a $50K+ package. Each D2D link adds a high-speed interconnect test pattern, and per-die screening multiplies machine hours.

3. HBM stacking. HBM3E ships as 12-Hi stacks with a 1,024-bit interface and ~1,024 TSV connections per layer. Every DRAM die is tested individually, then the full stack is validated as a Known-Good-Stack-Die (KGSD). HBM3E test time runs roughly 2× HBM2E. HBM4 (16-Hi, 2,048-bit) lengthens it again.

4. Power and thermal. Top-line AI accelerators draw 700–1,200W. Test sockets must dissipate that thermal load while maintaining clean signal integrity, and the chip must be exercised under representative AI workloads — not just static patterns — to surface latent defects. This forces coolant-integrated handlers and longer at-load test intervals.

The aggregate effect: cost of test as a percentage of chip ASP has roughly tripled on leading-edge AI parts — from historical levels of <2% toward 5–7% on the most complex packages. The customer can’t do anything about it. Yield economics on a $30–70K chip mean any defect that escapes test costs orders of magnitude more than the test itself. The bill scales linearly with units shipped, and it accrues almost entirely to the two companies whose platforms are qualified for the job.

A duopoly with structural lock-in

Advantest’s V93000 is the qualified SoC tester at Nvidia, AMD, and the merchant ASIC supply chain (Broadcom-Google, Marvell-Meta/Microsoft). Teradyne’s UltraFLEX dominates Apple silicon and the broader mobile/auto base.

The lock-in comes from the test program itself — a software artifact representing 6–18 months of engineering work, deeply integrated with the chip’s design flow, that gets re-used across every wafer for the chip’s full lifecycle. Once a fab and OSAT qualify a tester, replacement risk approaches zero.

That’s why the recent share dynamics matter so much: Advantest gained 10 percentage points of SoC tester share in CY25 alone — from 56% to 66% — by sitting inside the AI accelerator supply chain at a moment when AI accelerator volume is the only segment of semiconductors growing meaningfully. Total tester market share moved from 58% to 65%. Teradyne gave up incremental SoC share but is monetizing the same demand wave through its broader portfolio: system-level test, photonics, robotics-assisted manufacturing.

The financial signature

The technical bottleneck shows up cleanly in the sales and margin figures.

Advantest FY25 (year ended March 2026): operating margin moved from 29.3% to 44.2% — a 1,490bp expansion in twelve months on revenue of $7.5B, +47% YoY in USD terms. Operating income roughly doubled to ~$3.3B. Q4 ran at 46.7% operating margin on 67.4% gross.

Teradyne Q1 2026, annualized: the run-rate is $5.1B in revenue, ~$1.9B in operating income, and $10.24 in non-GAAP EPS — versus a $2.7B revenue / ~$0.6B op-income / $3.00 EPS run-rate one year earlier. Non-GAAP operating margin moved from 20.5% to 37.5%, a 1,700bp YoY expansion.

Both companies are now operating inside or above their long-term target margin bands. This isn’t volume leverage alone — it is pricing on complex testers in short supply, the financial fingerprint of a duopoly with a captive AI accelerator customer base.

Where the next leg of growth comes from

The interesting part of the story isn’t ATE itself. It’s the adjacent test layers that have emerged because traditional ATE no longer catches every defect that matters in an AI workload.

• System-Level Test (SLT). Exercises the chip in conditions close to its end-use — running real workloads, at full thermal load, with real power delivery — to catch latent defects that ATE patterns miss. SLT economics are different (more chambers, longer test times per unit) and the addressable market is multi-billion incremental on top of the core ATE TAM. Teradyne explicitly anchored its $6B revenue target model around the “wafer-to-AI-datacenter” framing. Advantest is positioning through M&A in system-level and silicon validation.

• Silicon photonics and co-packaged optics (CPO). As Nvidia, Broadcom, and the hyperscaler ASIC supply chain move to optical interconnects between accelerators and switches, every photonic die requires both electrical and optical test — a category that didn’t materially exist three years ago. Advantest received its first high-volume ATE order for silicon photonics in CY25. Teradyne launched the Photon100 platform specifically for this.

• Probe cards, contactors, and test interface boards. Teradyne owns ~10% of Technoprobe. Cohu and FormFactor sit alongside as picks-and-shovels exposure.

• AI-driven adaptive test. Uses ML to reduce test time on known-good silicon. Counterintuitively margin-positive for the duopoly because it sells as software-attached value on the same hardware base.

Outlook and risks

Capacity is the binding constraint, not demand. Advantest is ramping to 10,000 SoC test systems per year — up from a 7,500 target — to keep pace with a known product pipeline: Blackwell Ultra (B300) and AMD MI400 in 2026, custom AI silicon from Broadcom, Marvell, and Alchip stacking through OSAT lines, and HBM4 (16-Hi, 2,048-bit) qualifying for production. Each generation lengthens the test program. Teradyne is signaling 55–60% first-half revenue linearity — the order book is already loaded.

The market backdrop is just as durable. Advantest forecasts the SoC tester market at $8.7–9.5B in CY26 versus $6.9B in CY25, and the merchant AI ASIC supply chain (Broadcom-Google, Marvell-Meta/Microsoft, AWS Trainium) keeps multiplying tester demand because each new custom silicon program qualifies a fresh test program. The four adjacent product surfaces above — SLT, photonics, probe cards, adaptive test — extend the runway by another half-decade beyond the core ATE box.

Risks live on the product roadmap, not the financial model. In-sourcing by Nvidia or TSMC has been explicitly rejected on engineering ROI grounds. Chinese indigenous ATE remains generations behind in leading-edge SoC. Customer concentration in the AI OSAT supply chain — most acute for Advantest — is the largest single tail risk; a hyperscaler capex digestion cycle is the scenario worth tracking into 2027.

Net: the cleanest non-Nvidia exposure to AI accelerator unit volume — a duopoly with order-of-magnitude growth in test surface area, three multi-billion adjacent product surfaces, and the engineering switching costs to defend share through the next decade..

The thing that collapsed is called the KV cache. Understand how it works, and the next five years of AI economics, hardware roadmaps, and infrastructure margins start to make perfect sense.

Two cost lines

Running an AI model (inference) comes down to two cost lines:

• The Model Weights (Fixed Cost): These are the model’s “brains,” paid for during the massive training runs and amortized across every request the model ever serves. With enough traffic, weights are essentially free.

• The KV Cache (Variable Cost): This is the model’s “short-term memory.” It is paid for in High Bandwidth Memory (HBM)—the ultra-fast, ultra-expensive RAM that sits right next to the GPU. This cost is incurred per user, per word, on every single request. It does not amortize. It scales linearly.

This second line dictates the unit economics of AI. It is also the line frontier labs talk about least, because it destroys their published “price-per-token” the moment a user’s prompt gets too long.

What the cache actually is

Transformers generate text one word (token) at a time. To write the next word, the AI has to look back at every previous word in the conversation. Re-reading the entire conversation from scratch every single step would be incredibly slow, so the model saves its notes in a “Key-Value” (KV) cache in the GPU’s memory.

This cache grows with every single word you type or the AI generates. For a large model like Llama 3 70B, a 128,000-token conversation eats up ~40 GB of memory per user. Multiply that by concurrent users hitting the same server, and you hit a memory wall long before you run out of raw computing power.

The bottleneck is bandwidth, not compute

People who grew up in the software era expect processor speed (compute) to be the bottleneck. It isn’t.

Modern GPUs like the NVIDIA H100/B200 are incredibly fast calculators, but their memory bandwidth can’t keep up. The math processors spend most of their time sitting idle, waiting for the short-term memory (the KV cache) to be fetched and delivered.

The math is brutal. When generating a response for a long document, the GPU is essentially reduced to a giant memory streamer with a math coprocessor stapled on. The speed limit is entirely dictated by how fast data can move through the HBM. This is why every 2026 accelerator roadmap is aggressively focused on memory bandwidth, not raw processing speed.

How Long Context Broke the Pricing Model

AI companies publish a single price per million tokens. That price is modeled against short interactions (4K to 8K tokens) where the KV cache is small enough to be a rounding error. Run that same model at 128K tokens, and the cost structure is completely unrecognizable.

Walk through the numbers: An H100 GPU holds the model’s weights and uses whatever memory is left over for the users’ KV cache. At an 8K context, a server might support ~190 concurrent users. At a 128K context, it supports one or two. It’s the same GPU, running the same model, charging the same nominal price—but the actual cost to the provider just skyrocketed.

This is why providers quietly changed their pricing in 2025. “Cached” tokens—where short-term memory is reused across similar requests—now cost a fraction of fresh tokens. The discount looks generous to developers, but it’s actually a confession from the AI labs about their underlying cost curve.

Agents will make this worse. Autonomous AI agents don’t have clean, reusable memory. They constantly replay long, mutating histories as they use tools and browse the web. Anyone modeling enterprise AI margins based on simple chatbots is in for a shock when traffic shifts to agents.

Architecture is outrunning hardware

Look at where the recent gains came from.

• Hardware: An H100 to B200 upgrade yields 2.4x more HBM and 2.4x more bandwidth.

• Software: The algorithmic stack has compounded efficiency by roughly 50–100x on the same silicon.

  • MHA —> GQA (8:1): Grouped-Query Attention reduces memory footprint by forcing multiple “query” heads to share a single “key-value” memory head, cutting the cache size by a factor of eight with almost no drop in intelligence.

  • GQA —> MLA (DeepSeek V2/V3): Multi-head Latent Attention massively compresses the memory cache into a tiny, dense mathematical space and decompresses it only when needed, yielding extreme cost savings with no reported quality loss.

  • FP16 —> INT4 KV quantization: Quantization physically shrinks the memory by rounding the high-precision decimal numbers down to smaller, 4-bit integers, trading a microscopic fraction of accuracy for a 4x reduction in size.

  • PagedAttention (vLLM): Instead of shrinking the data itself, PagedAttention eliminates wasted empty space in the GPU by breaking the cache into flexible blocks, acting like a highly efficient hard drive defragmenter for AI.

  • Prefix caching: Prefix caching saves the memory state of frequently used text—like system instructions or shared documents—so the model can instantly reuse it across hundreds of users instead of recalculating it from scratch.

  • Sliding window / hybrid SSM: These alternative architectures cap memory usage at a fixed, constant size by either only remembering the most recent conversation history or compressing the past into a static state, entirely eliminating the linear growth of the cache.

DeepSeek’s Multi-head Latent Attention (MLA) is arguably the most important architectural innovation of the last two years. It essentially compresses the KV cache and decompresses it on the fly, cutting memory usage by over 90% with no loss in quality. Every major AI lab will be forced to ship a version of this in 2026. Leaving that kind of cost advantage on the table while DeepSeek undercuts API prices is not a viable business strategy.

Where the capital flows

High Bandwidth Memory (HBM) is the actual scarcity in AI infrastructure right now. NVIDIA’s pricing power is bounded less by chip manufacturing limits than by how much HBM they can physically buy from suppliers like SK Hynix and Samsung. In 2026, the Big 3 Memory Players will sell over $70B of HBM at >60% margins.

This dynamic creates three durable opportunities and one trap:

• Opportunity 1: Alternative Silicon. Startups like Cerebras, Groq, and Etched are betting that the right chip for running AI isn’t an HBM-starved GPU, but a specialized chip designed specifically to eliminate memory bottlenecks. If they capture even a narrow slice of the market—like high-speed AI agents—NVIDIA’s monopoly on serving AI will crack.

• Opportunity 2: Memory-Smart Infrastructure. Differentiation in AI hosting has moved from “we have GPUs” to “we manage memory states better than you.” The open-source infrastructure stack (vLLM, continuous batching, prefix caching) is evolving to manage memory intelligently. Running this effectively at scale provides a massive, quiet moat for infrastructure companies.

• Opportunity 3: New AI Architectures. We are seeing the rise of new models (like Mamba or Jamba) that ditch the traditional Transformer architecture entirely. Instead of a KV cache that grows linearly with every word, they use a constant, fixed-size memory state. If these models hit the same quality as OpenAI or Anthropic, the expensive, memory-heavy GPU stack becomes optional for a huge chunk of workloads.

• The Trap: “GPU-as-a-Service.” Renting out H100s behind a simple API is a shrinking-margin business. Customers can buy raw compute from twenty different providers. The real profit margin lives in the software layer that manages the memory and routes the traffic—neither of which a basic GPU broker controls.

The decisive number

If you take one thing away from this post, make it this: When an AI is reading a long document, the vast majority of its time is spent just moving short-term memory through the hardware. The compute is already paid for. The customer is paying for a memory traffic jam.

That memory traffic is the variable cost of intelligence. It is the line item that dictates who enjoys 80% profit margins in AI, and who scrapes by with 30%. And it is the exact bottleneck that the next wave of tech innovation is coming to solve.

The companies that figure out how to bypass, compress, or outsmart the KV cache will own the unit economics of AI for the rest of the decade. Everyone else is just renting GPUs.

What replaces the single chip is a routed inference stack: each request flows through a FLOPs-rich accelerator for prefill, hands the KV cache to a memory-bandwidth-rich accelerator for decode, and streams the output back. The diagram below is the architecture; the rest of this piece is why the physics demand it and what the market consequences are.

The Two Phases

LLM generation is autoregressive: every output token depends on every token that came before it. That dependency creates two computational regimes.

Prefill processes the input prompt. Every prompt token is known up front, so the model computes their key/value vectors in parallel — one large matrix multiply that builds the KV cache. Compute-heavy. FLOPs-bound. High arithmetic intensity, on the order of 100–400 FLOPs per byte read.

Decode generates output tokens one at a time. Each step streams the full model weights and the entire KV cache from memory to multiply against a single activation vector, then appends one new K/V pair to the cache. Bandwidth-heavy. Low arithmetic intensity, roughly 1–2 FLOPs per byte.

Same model. Same hardware. Two different workloads.

The Metrics That Decide Deployment

Two latencies decide whether an inference deployment is usable.

• Time to first token (TTFT) is set by prefill. Interactive UX breaks above roughly 3 seconds.

• Time per output token (TPOT) is set by decode. Interactive targets sit at 100–300 ms per token, at minimum matching reading speed.

API pricing reflects the asymmetry. Output tokens cost meaningfully more than input tokens at every major provider — GPT-4o lists $2.50 per million input tokens against $10.00 per million output tokens, a 4x premium on the memory-bound phase. Decode is where the margin lives.

Why One Chip Can’t Do Both Well

Modern GPUs are designed around HBM-fed tensor cores. Prefill uses both. Decode uses neither well — the tensor cores idle while HBM saturates streaming weights for a single output token at a time. Plot it on a roofline and the two phases sit on opposite sides of the knee.

Inference engines have spent the past two years working around this. vLLM’s chunked-prefill scheduler interleaves prefill chunks with decode steps so compute and memory bandwidth saturate simultaneously on the same GPU. TNG Technology Consulting reports the technique increased total token throughput by approximately 50% in their production vLLM deployment serving Llama-3.1-8B (source). The lesson is structural: even on one chip, the operator gets paid to treat prefill and decode as different jobs.

The silicon-level unbundling is the same trick at scale.

Nvidia + Groq: FLOPs Buys Bandwidth

Nvidia owns prefill. Hopper and Blackwell are HBM-rich and FLOPs-rich, and they run the dominant share of frontier-model context processing. They do not own decode economics. HBM is supply-constrained — three vendors (SK Hynix, Samsung, Micron) — and decode burns bandwidth that could otherwise serve training.

The relevant Nvidia HBM specs:

Each generation buys more bandwidth, but the underlying architecture is still HBM-fed tensor cores. Decode still loads the full weight matrix from external memory for every token.

Groq is the architectural opposite. No HBM. The LPU runs entirely from on-die SRAM, which is faster per byte than HBM by at least one order of magnitude and is not bottlenecked by the HBM3e supply chain. The chip does not win on raw FLOPs. It wins on bandwidth-per-dollar, where bandwidth is the constraint — i.e., decode.

Acquiring Groq lets Nvidia route prefill to Blackwell and decode to LPUs over high-bandwidth fabric. The HBM bus stops being the bottleneck on output-token economics. Nvidia’s HBM exposure on inference workloads tightens. The strategic prize is freeing HBM supply for training, where it has no substitute.

Cerebras + AWS: SRAM Meets Trainium

Cerebras is the wafer-scale extreme. The WSE-3 is a single 46,225 mm² die — roughly 57x the silicon area of an H100 — with 44 GB of on-chip SRAM. A 70B-class model and its KV cache fit on-die. Decode runs at SRAM speed. Cerebras has built its commercial pitch around sustained tokens-per-second on memory-bound inference, and the architecture maps directly onto the decode workload.

What Cerebras lacks is fleet-scale prefill capacity. CS-3 systems are scarce, capital-intensive per unit, and optimized for memory-bound work. Long-context prefill — increasingly the dominant workload as agentic and reasoning models drive context windows toward 1M tokens — needs cheap, abundant FLOPs.

AWS Trainium fills the gap. Trainium2 is a high-FLOPs, HBM-equipped accelerator built for compute-bound work. The integrated Cerebras–AWS stack runs prefill on Trainium fleets, hands KV state to Cerebras CS-3 systems, and serves decode at wafer-scale bandwidth. AWS monetizes Trainium silicon on a workload where Nvidia GPUs already dominate. Cerebras gets the prefill volume the WSE was never designed to deliver.

Mirror-image strategy to Nvidia/Groq, same underlying thesis.

The Economics

Three forces make the unbundling structural rather than a clever optimization.

1. Workload mix is shifting toward decode. Reasoning and agentic models output substantially more tokens per request than chat-style models did 18 months ago. A growing share of frontier-model inference revenue comes from decode rather than prefill.

2. HBM is the binding constraint on decode capacity. HBM3e production runs near full utilization, with 2026 supply already pre-allocated. Any decode workload that migrates to SRAM-based silicon frees HBM for training and prefill, where the alternatives are worse.

3. Decode carries the margin. At a 4x price premium over input tokens, decode is the gross-margin lane. Operators serving decode on lower-cost silicon capture the spread directly. A disaggregated stack lowers cost per output token versus a single-architecture deployment — the exact figure depends on workload mix, but the sign is unambiguous.

What This Changes

1. The HBM oligopoly’s pricing power weakens at the margin. Decode migration off HBM-dependent silicon slows HBM3e demand growth and reshapes HBM4 timing assumptions.

2. Inference orchestration becomes a control point. The router that decides which phase runs on which silicon — and tunes batch size, chunk size, and KV-cache transfer policy — is the new margin layer. The technical primitives already exist inside vLLM (chunked prefill, prefix caching, KV-cache offloading). The step from “interleave on one GPU” to “route across heterogeneous accelerators” is small. Nvidia (Dynamo, post-Groq), AWS (Bedrock + Trainium + Cerebras), and independent inference platforms (Together, Fireworks, Baseten) will fight for it.

3. Capital intensity per token declines. A unified GPU stack overpays for FLOPs on decode and overpays for HBM on prefill. The disaggregated stack pays for each only where it is consumed.

4. New silicon entrants get a clearer wedge. The single-architecture incumbency advantage erodes when the workload itself is two distinct workloads. Memory-bound silicon (Groq, Cerebras, d-Matrix, Tenstorrent) gets a defensible decode lane. Compute-bound silicon keeps prefill but loses its decode rent.

Bottom Line

Prefill and decode are different physics. Different roofline regimes. Different cost structures. Different margin profiles. Inside a single GPU, scheduler-level disaggregation already pays. Across silicon, Nvidia/Groq and Cerebras/AWS are the same trade — pair a compute engine with a memory engine, route the workload, capture the spread.

The single-accelerator era is over. The routing layer owns the margin.

Security has the same structure. Did the exploit land? Did the agent reach the flag? Did the defender detect and contain it before exfiltration? Binary, automatically verifiable, and infinitely repeatable in a sandbox. The missing ingredient has been the environment infrastructure to produce these signals at scale. That is now being built. Anthropic is already publishing research with companies like Incalmo on breaching systems with AI.

The Drone Parallel

Cost asymmetry has become the defining strategic problem of kinetic warfare:

A $20K munition forcing a $4M interceptor breaks defense economics. The industrial response — Anduril, Epirus, directed energy programs at the primes — is fundamentally a race to push defense cost-per-engagement toward $1–$10 (lasers, microwave, EW).

Cyber is the same problem with worse asymmetry:

• AI attacker compute per engagement: $10–$100

• Average enterprise security spend: $5M–$50M+ annually

• Average breach cost (IBM 2024): $4.88M

• Open-source attacker tooling: $0

How do you stop a $10 autonomous exploit without spending $1M per incident? That is the question the next decade of security spend will be organized around.

What Is Different From Drones

Three things make the cyber version harder:

1. Software has no kinetic bottleneck. Drone attackers need factories. Cyber attackers need a checkpoint and GPUs.

2. The defense surface is unbounded. Airspace is finite. Modern enterprise attack surface — identities, SaaS, supply chain, model weights, agent tools — is not.

3. Attribution is weak. Deterrence-by-punishment underwrites kinetic doctrine. It does not apply in cyber.

The implication: the AI defender must be cheaper, faster, and more autonomous than its kinetic equivalent. A human SOC analyst at $200K fully loaded handling 10–20 investigations per day cannot match an attacker running thousands of episodes per hour.

The Training Bottleneck

You cannot RL-train an agent in production. The analog to a unit test or a proof checker is a sandboxed environment with realistic topology, verifiable objectives, adversarial structure, and parallel scale — thousands of episodes per hour. This is the picks-and-shovels layer for security RL. Whoever supplies the environment captures economics from every downstream model.

Implications

• Offensive AI commoditizes faster. Assume open-source parity with commercial tooling within 24 months.

• Defensive AI consolidates around whoever owns the best environment. That is where the training data lives.

• Legacy SIEM/SOAR is structurally mispriced. A 6 figure/year log platform cannot defend against a $10/hour autonomous attacker. Expect a generational vendor reset within 36 months.

• Capital flows in order: environments first, defender models second, deployed agents third — mirroring how drone-defense capital rotated from platforms to payloads to fire control.

Summary

Verifiable-reward RL fell math and code in under 24 months. Security has the same reward structure and a larger economic surface. The drone analogy gives the cost curve; the lesson is that whoever industrializes the reward environment first determines who wins the next decade of the attacker-defender game.

AI inference workloads require 10,000–1,000,000x the compute of a comparable SaaS transaction, depending on model size and task complexity. This shifts value from software toward compute supply chain owners, compressing gross margins at the application layer. Hyperscalers are spending a combined $700B in 2025–2026 capex — roughly 5x the 2022 baseline — because AI-native workflows represent a step-function increase in compute demand over the SaaS baseline they replace.

What determines the actual production of compute-hours? Most discussion focuses on GPUs. But compute-hours are the output of a system with 7–10 interdependent inputs. Each layer is tightly coupled. When constraints stack, supply elasticity collapses.

Key Trends

1. Nvidia sells full racks, not GPUs. The unit of compute has shifted from chip to rack. GPU count per rack moves from 72 (Hopper) to 576 (Rubin) — 8x in one generation. Rack-level memory is scaling faster: GB200 NVL72 at 30 TB HBM3e, GB300 NVL72 at 40 TB, Vera Rubin NVL144 projected at 100 TB — roughly 3x within the Blackwell generation alone. Per-GPU memory follows the same trajectory: H100 at 80 GB HBM3, B200 at 192 GB HBM3e (2.4x in a single generation), with HBM4 at Rubin pushing further. Every layer of the stack — power, cooling, interconnect, cabling — gets redrawn when the reference architecture moves.

2. The interconnect question is unresolved. Scale-up (within rack) and scale-out (rack-to-rack, pod-to-pod) have diverging physical layer requirements. Copper and AECs remain competitive at short reach on cost, reliability, and power. Co-packaged optics improves bandwidth density at distance but carries a power premium over copper today. Where the crossover lands determines which interconnect vendors are structural winners.

3. Memory is a primary constraint. HBM capacity growth is outpaced by model size and context length expansion. Supply is concentrated across three vendors. HBM accounts for roughly 35–40% of AI server bill-of-materials cost, making memory economics a system-level concern, not a component-level one. Any component that reduces memory pressure — near-memory compute, KV cache compression, sparsity — commands disproportionate ROI in the current supply environment. Per-GPU memory progression: H100 (80 GB) → H200 (141 GB, 1.76x) → B200 (192 GB, 1.4x) → Rubin (288 GB est., 1.5x) — roughly 3.6x over six years.

4. ASICs are in production, not pilot. Google TPUs and AWS Trainium are at scale. Each requires its own compiler toolchain, interconnect topology, and systems integration. The infrastructure stack can no longer be optimized solely around NVLink/NVSwitch.

5. Energy is an architectural input, not an operational variable. Rack power density has increased from 10–20 kW (CPU era) to 100–130 kW (GB200), with 200+ kW expected at Rubin densities. Optical transceivers consume roughly 2–3x more power per port than copper at equivalent bandwidth — a material tax at 200+ kW/rack. Decisions that appear to be performance tradeoffs are simultaneously energy budget decisions.

Scale-Up Clusters: Why NVIDIA’s Architecture Wins on Durability and Efficiency

GPU density per rack has increased roughly 6x from 2021 to 2026 — from 8–16 GPUs per rack to 72+ in the GB200 NVL72, with the NVL144 and NVL256 roadmap targeting continued doubling. This is a systematic architectural shift, not incremental improvement, and it compounds into durable cost and performance advantages.

The Communication Hierarchy: Why Staying In-Rack Is Structurally Cheaper

Cluster communication cost and latency increase sharply as data crosses physical boundaries. The hierarchy, from lowest to highest cost: (1) in-package and NVLink — sub-microsecond latency at 900 GB/s per GPU in the NVL72; (2) rack-scale interconnect — copper-viable but bandwidth-constrained; (3) cross-rack networking — where latency and cost increase nonlinearly. Every inter-rack hop adds microseconds of latency, consumes optical transceiver power (~2–3x per port versus copper at equivalent bandwidth), and introduces fabric congestion risk. Larger in-rack clusters reduce total communication overhead and deliver lower cost-per-token at scale. The math is structural.

NVIDIA’s Structural Advantage: NVLink as a Moat

NVIDIA is the only vendor currently able to deliver a validated, high-density scale-up cluster at rack scale. The NVL72 ships with 900 GB/s GPU-to-GPU bandwidth via NVLink 4, 30 TB of pooled HBM3e memory, and a single-rack power envelope of ~120 kW within NVIDIA's reference architecture. Competitors must traverse PCIe or Ethernet for equivalent multi-GPU communication — 10–20x lower bandwidth per link than NVLink at equivalent rack density. AMD's Infinity Fabric does not scale beyond 8 GPUs without external switching. Google TPUs and Amazon Trainium are ASIC-scale designs without a comparable proprietary high-bandwidth fabric. NVIDIA's inference cost-per-token advantage at 72-GPU density is structural, not a function of current market position.

Efficiency and Durability Implications

First, eliminating inter-rack hops removes optical transceiver power overhead — material when each 800G optical port draws ~10 W versus ~4 W for equivalent copper.

Second, unified HBM memory pools across 72 GPUs allow workloads to avoid KV cache eviction and model sharding penalties that dominate inference latency at smaller cluster sizes.

Third, higher GPU utilization: the GB200 NVL72 sustains higher model flops utilization on large-batch inference workloads than disaggregated clusters of equivalent total GPU count, because all-reduce operations stay within NVLink rather than crossing fabric.

NVLink all-reduce is 4–8x faster than InfiniBand at equivalent cluster size. As inference workloads scale, the performance gap between NVLink-based and non-NVLink architectures widens. The NVL72 roadmap to NVL144 and NVL256 compounds CUDA ecosystem lock-in with physical architecture lock-in.

Investment note: The industry-wide push toward larger scale-up clusters requires exponentially more cables, higher signal integrity requirements, and greater pressure on copper and optical interconnect at short reach as GPU density per rack increases. This trend is net positive for high-bandwidth interconnect specialists addressing signal integrity challenges at 72+ GPU densities — including Attotude.

Where we are today: massive shortages

The underappreciated risk is that a single bottleneck shuts down the entire system. Current materials under extended constraint: copper, aluminum, indium phosphide; power and electrical equipment; laser components; HBM memory; CoWoS packaging capacity at TSMC. Most are booked out 2+ years. Removing one constraint shifts pressure to the next layer — which is why aggregate GPU supply numbers mask the actual delivery reality.

Structural Constraints — Severity, Bottleneck, and Timeline

Core Thesis

The AI infrastructure stack is not just GPU-constrained — it is system-constrained. Removing one bottleneck shifts pressure to the next. Companies that own chokepoints in this stack — advanced packaging, power delivery, optical interconnect, laser components — hold structural leverage analogous to what ASML holds in semiconductor manufacturing.

Four structural forces are reshaping the stack:

1. Slow-scaling inputs versus fast-scaling demand. Power generation and high-voltage transmission scale on decade timelines. Compute demand scales on 12–18 month cycles. The gap between these curves is the primary investment surface.

2. Layer-specific bottlenecks force architectural transitions. When a layer hits a physical limit, the supplier map is redrawn. Copper-to-optical in networking is the live example — CPO share expected to rise from near zero to ~36% of addressable ports.

3. Silicon is fragmenting. AMD, Google TPUs, Amazon Trainium, and OpenAI custom silicon are eroding NVIDIA’s training monopoly. Value migrates toward scarcer upstream inputs when the compute layer commoditizes. NVIDIA remains the only vendor selling the full rack.

4. Compute is geography-bound. Power availability, transmission access, and regulatory environment are making AI infrastructure decisions location-specific. Texas has more planned data center capacity than any other state. Site selection is now a primary capital allocation decision, not a secondary one.

Recent M&A

The M&A velocity in optical and interconnect is the strongest forward indicator of where supply constraints are most acute.

• Groq —> Nvidia $20B, inference chip

• Celestial AI → Marvell ~$3.25B

• Optical connectivity | Enfabrica → NVIDIA ~$900M

• Ayar Labs ($1B+ Series D) — likely next optical exit

$4.15B in optical-adjacent exits in under 12 months. NVIDIA’s $2B strategic investment in Lumentum and Coherent (March 2026) is a direct acknowledgment that laser supply — specifically electro-absorption modulated lasers inside every 800G and 1.6T transceiver — is a compute constraint. Lumentum controls 50–60% of global EML supply.

Public Market Exposure — Where to Express Today’s Needs

Five Key Structural Observations

1. Optical fabric is the highest M&A velocity category.

$4.15B in exits in <12 months. Ayar Labs ($1B+ Series D) is the next likely event. The laser layer is the least visible and most acute constraint within optical: electro-absorption modulated lasers (EMLs), the component inside every 800G and 1.6T transceiver that converts electrical signals to light, are undershipping demand by ~30%, with all capacity locked under long-term agreements and incremental units priced at a premium. Lumentum controls 50–60% of global EML supply; Nvidia’s $2B strategic investment in Lumentum and Coherent (March 2026) is a direct acknowledgment that laser supply is a compute constraint.

2. CoWoS is the rate-limiting step for everything upstream.

GPU supply, custom ASICs, and AI accelerators all flow through TSMC CoWoS at ~20K wafers/month. Independent packaging alternatives and panel-level interposers are the highest-leverage supply chain investment in this cycle.

3. Power is a software opportunity.

Grid interconnection, energy OS (1,700+ utilities, multiple types of power <> data center), and workload-aware power management are software wedges into a hardware-constrained market. Vertiv captures the hardware layer; the software orchestration layer is open.

4. Inference economics are separating from training economics.

Training is NVIDIA’s domain. Inference is not. Cost/token, memory efficiency, and latency matter, and they are not optimized by training-oriented GPU architectures. Critically, LLM inference, image generation (fal.ai, Flux), and video generation (Runway, Sora) have distinct compute profiles — model-type-specific accelerators may be a more tractable bet than general inference silicon.

5. NVIDIA is vertically integrating; build around it.

Enfabrica (fabric), Mellanox (networking), Cumulus (switching). Companies must build on technology NVIDIA does not control: Ethernet AI fabric, RISC-V, photonics, advanced packaging. Partner, don’t compete.

The severity of the supply-demand mismatch is visible not just in procurement data but in production reliability. Anthropic — among the most compute-intensive AI companies in the world — operates below 98% uptime, a figure that would be disqualifying for any enterprise software vendor. AWS S3 targets 99.99%; Salesforce and Workday routinely exceed 99.9%. Anthropic’s reliability ranks among the lowest of any major software service, not due to engineering failure but due to compute scarcity. It’s reported that much training resources in 2026 will be diverted to keep inference online, ultimately slowing model releases.

Despite concerns that further generations of Nvidia hardware would cause heavy depreciation on older models like H100s, prices have hardly come down and even become more scarce over time:

Frontier AI labs are diversifying their compute base. The most consequential shift is not a single provider’s growth — it is the pattern of AI labs actively decoupling from single-cloud dependency. OpenAI is diversifying away from exclusive Microsoft reliance toward OCI, which absorbed a material portion of its recent backlog growth. Anthropic is expanding beyond AWS and GCP into FluidStack, Crusoe, and Oracle. These are strategic decisions by the two largest AI labs to distribute infrastructure risk across providers. OCI’s backlog growth is real, but its concentration in OpenAI creates bilateral risk: OpenAI has extended itself aggressively across multiple infrastructure commitments and may restructure obligations if its own revenue trajectory compresses.

Even Microsoft cannot keep up. Azure’s $625B AI-era backlog represents +1,150% growth over its pre-AI baseline — and it is still not enough. Microsoft has spent the past year signing capacity deals with CoreWeave, Crusoe, Nebius, and NScale, sourcing a rapidly growing percentage of incremental compute from third parties. This is not a strategic pivot — it is a supply constraint forcing structural dependency.

The neoclouds are real, though still small. CoreWeave, Nebius, NScale, FluidStack, and Crusoe entered the table from zero, combining for $131B in new commitments — enterprise-scale, multi-year GPU reservations from customers and hyperscalers alike who cannot build fast enough to serve demand internally. In practice these remain small relative to the $100B+ investments made by hyperscalers. Today neoclouds represent 10–20% of capex in AI data centers; hyperscalers account for 80–90%.

The Capital Ceiling

The four largest hyperscalers — Microsoft, AWS, Google, and Meta — are collectively committing approximately $700B in AI infrastructure capex in 2026, a figure that approaches or exceeds 100% of their operating free cash flow and creates negative overall free cash flow. Boards impose discipline; buybacks, dividends, and core business investment compete for the same capital.

The hyperscalers will not reduce capex programs, but they are unlikely to materially exceed the FCF ceiling on a sustained basis. The marginal dollar of new capacity will not come from Microsoft or AWS — it will come from neoclouds financed by private credit, infrastructure funds, and sovereign capital. They entered the commitments table from zero not because hyperscalers are retreating, but because hyperscalers cannot grow fast enough to serve demand at the margin.

In less than three years, Microsoft transitions from a fully self-supplied infrastructure operator to one sourcing roughly half its incremental compute from third parties. For Microsoft, CoreWeave is not a partner — it is a crutch. For CoreWeave, Microsoft is not a customer — it is validation that the largest infrastructure spender in the world cannot meet its own demand.

Why This Is Structural, Not Cyclical

Compute-hours are produced by a chain of interdependent inputs. The binding constraints have migrated from silicon to physical infrastructure: power availability, high-voltage transmission, transformer manufacturing (lead times: 2–4 years for large units), cooling, and grid interconnection. Demand scales on 12-to-18-month cycles; electrical infrastructure scales on decade-long ones.

That mismatch is structural scarcity. When one constraint is relieved, pressure shifts to the next layer serially — GPU supply exposes the power bottleneck; power exposes transformer lead times. The result is geography-bound compute: AI infrastructure is forced into specific regions by power availability, land, water access, and regulatory environment. OCI and CoreWeave have been structurally more aggressive on power procurement than AWS and Azure at the margin, and the backlog data reflects it.

Networking follows the same pattern: copper interconnects are hitting limits at 10,000-plus GPU cluster scale, forcing a transition to co-packaged optics (CPO) that collapses a supplier tier and elevates a new set of component manufacturers into structurally critical positions.

Implications

Every hyperscaler is growing — and none are growing fast enough. These are extraordinary growth rates by any historical standard. Yet neocloud backlog emerged from zero in the same period. The demand signal is not that hyperscalers are failing; it is that AI compute demand is scaling faster than even trillion-dollar balance sheets can absorb. GCP’s +382% growth suggests Google is executing better on AI-native positioning than its installed base would imply.

OCI is the most important cloud story to watch — and possibly the most exposed to ROIC failure before any neocloud is. Oracle recently eliminated approximately 30,000 employees, inconsistent with scaling into a large build commitment. A reported Texas site failure raised delivery timeline questions. Most consequentially, Larry Ellison has reportedly pledged Oracle equity as collateral against personal debt tied to a significant media acquisition, creating forced-selling exposure and board independence risk at exactly the moment the company requires sustained capital discipline.

Oracle (ORCL) — LTM Cash from Operations: $22.3B / CAPEX: $35.5B / Free Cash Flow: -$13.2B. Source: Bloomberg

The conventional risk framing treats neoclouds as the fragile capital structures and hyperscalers as the durable ones. In OCI’s case, that framing may be inverted. A purpose-built neocloud with no founder pledge overhang, no legacy workforce to restructure, and no unrelated media liability may be a more reliable 5-year counterparty. OCI’s backlog growth is real; whether Oracle has the organizational and financial architecture to deliver it is an open question.

Neoclouds are the marginal capacity provider. CoreWeave’s $41B, Nebius’s $20B, and Crusoe’s $20B are the overflow valve for a market growing faster than any single balance sheet can absorb. The neocloud capital structure — private credit, infrastructure funds, sovereign capital — is purpose-built for exactly this role.

Regionalization is the next structural theme. AI infrastructure is geography-bound by power availability, data sovereignty, and national industrial policy. NScale ($14B valuation) is the primary neocloud partner for Microsoft and OpenAI in Europe. Neysa is executing the same playbook in India. The Middle East and Latin America are next — sovereign AI programs, abundant energy, and governments subsidizing local infrastructure to avoid permanent compute dependency on US providers. Expect regional neocloud champions in each geography, underwritten by hyperscaler offtake agreements and sovereign capital.

Risks

Risk 1 — Proprietary Silicon at Scale

The one credible path by which hyperscalers recapture internal workloads from neoclouds is success with custom ASICs. AWS (Trainium/Inferentia) and Google (TPUs) are the only hyperscalers with viable proprietary silicon in production today — but at relatively small scale relative to their total compute footprint. Microsoft, Oracle, and Meta have no viable ASIC. OpenAI is working with Broadcom on custom silicon but is at minimum three years from meaningful production. Meta’s response is instructive: rather than build internally, it has committed to AMD at scale — reportedly one of the largest GPU procurement agreements in the MI300X generation — wagering that a third-party accelerator can substitute for internal silicon. Until AWS and Google demonstrate ASIC scale that materially reduces neocloud dependency, the ASIC threat remains a 5-year risk, not a present one.

Risk 2 — Demand Deceleration

The one scenario in which neoclouds face existential pressure before hyperscalers do is a rapid cooling of frontier AI demand — a world in which model capability improvements plateau, enterprise adoption stalls, or capital markets withdraw from AI infrastructure financing. Neoclouds carry the most concentrated exposure: GPU lease obligations are fixed, cost structures are capital-intensive, and there is no diversified revenue base to absorb a demand shortfall. Hyperscalers, by contrast, can redirect capacity to general cloud workloads and absorb losses within broader P&Ls. A demand collapse would reprice neocloud equity and debt simultaneously, and several newer entrants — underwritten on aggressive utilization assumptions — would face covenant stress before a single hyperscaler earnings call reflected the problem.

The evidence does not support near-term cooling. Anthropic and OpenAI are not experiencing outages because they are poorly run — they are experiencing outages because demand is outrunning every unit of capacity they can procure. These are rationing events in a supply-constrained market. GPU depreciation curves have proven slower than prior hardware cycles; H100 utilization rates have remained elevated and secondary market pricing has held above what prior GPU generations sustained at equivalent ages. Until the outage pattern reverses, a demand deceleration severe enough to threaten neocloud economics remains a tail risk, not a base case.

Project Rainier, Anthropic and AWS

Anthropic has built what is today the most diversified and cost-efficient compute architecture among frontier AI labs. OpenAI remains almost entirely dependent on Nvidia. Microsoft’s internal chip program is years behind schedule. The structural implications favor Anthropic on unit economics and negotiating leverage as inference workloads scale. While Anthropic has had so much demand, it has struggled with up-time — its long-term strategy is the most fundamentally resilient.

One important caveat up front: compute advantage amplifies model advantage; it does not replace it. If a competitor’s models are materially better, customers absorb the higher token cost. The argument here is not that Anthropic wins because of infrastructure. The argument is that equivalent model quality delivered at 30–60% lower cost per token is a compounding advantage — on margin, on training budget, and on the pace of iteration.

The Structural Divide

Every frontier model company faces the same binding constraint: various estimates put Nvidia at ~90%+ of the discrete GPU market and a dominant share of AI training accelerators deployed in the field, and that pricing power is real. H100 on-demand list pricing at major hyperscalers (AWS, Azure, GCP) peaked at $7–$8/hour; even after the 2025 oversupply correction drove spot and commodity-broker rates to $1.50–$3.00/hour, the published on-demand rate at hyperscalers still runs $6–$9/hour. Committed and reserved pricing falls somewhere in between — typically $2–$5/hour depending on contract length and volume — but requires capital-intensive multi-year lockup.

The question is not whether you can afford Nvidia GPUs. At scale, the question is whether Nvidia is the only lever you can pull, and what that dependency costs you annually.

There is a layer beneath the GPU pricing question that gets less attention: HBM supply. The real bottleneck in high-density AI clusters is not the accelerator die — it is High Bandwidth Memory, packaging, and power density. TSMC’s CoWoS packaging capacity is a genuine constraint, and Nvidia’s pricing power is partly a downstream effect of controlling where that supply gets allocated. Any chip independence strategy that bypasses this layer — custom accelerators still require HBM stacks — is incomplete. True chip independence means secured HBM allocation, packaging slots, and power-deliverable data center capacity; the die is just the most visible part. Anthropic’s TPU and Trainium commitments are meaningful partly because Google and Amazon have negotiated their own HBM supply chains, transferring that allocation risk away from Anthropic.

Anthropic answered the dependency question structurally. OpenAI has not.

AI LAB COMPUTE STRATEGY (March 2026)
──────────────────────────────────────────────────────────────────────────
Lab           Primary Stack          Diversification        Custom Silicon
──────────────────────────────────────────────────────────────────────────
Anthropic     TPU v5/v6 + Trainium2  Multi-hyperscaler      Operational now
              + Nvidia (research)    operational today

OpenAI        Nvidia H100/H200/B200  None at scale today    Broadcom: 2027+

Microsoft     Nvidia (dominant)      Maia 200: 2 sites      Maia 200: 2yr late
──────────────────────────────────────────────────────────────────────────

The strategic conclusion this table points to: the winning economic model is not chip ownership — it is deep integration into a hyperscaler’s silicon program. Labs that pursued partnership-driven vertical integration have operational compute today. Labs that pursued ownership are years behind.

Anthropic: Dual-Hyperscaler, Dual-Accelerator

Project Rainier

In October 2025, Amazon opened Project Rainier — a purpose-built AI compute campus in Indiana with an $11 billion infrastructure investment. The cluster runs on Trainium2, AWS’s custom AI accelerator, and operates at an estimated scale that makes it among the largest operational AI compute clusters in the world. Anthropic is its primary tenant; training Claude is the primary workload.

The economics are the point. Trainium2 runs at approximately half the price of comparable Nvidia H100 instances in sustained workloads. Effective committed cost drops to roughly $0.50/chip/hour — compared to $2–$5/hour for H100 on committed or reserved contracts at equivalent hyperscaler rates. Anthropic has reported 50% cost reductions and material throughput gains versus GPU configurations on specific training runs.

Amazon’s total investment in Anthropic now stands at $8 billion (following the initial $4 billion in 2023 and a further tranche in late 2024). This is not a vendor relationship. It is a strategic alignment that gives Anthropic priority allocation on the Trainium roadmap and infrastructure access at cost.

The Google TPU Deal

On October 23, 2025, Anthropic announced an expansion of its Google Cloud partnership committing to 1 million TPUv7 Ironwood chips — over 1 gigawatt of compute capacity — coming online in 2026. The total deal value is approximately $52 billion. The structure is split: 400,000 TPUv7 chips purchased directly from Broadcom for roughly $10 billion in finished racks (Broadcom CEO Hock Tan confirmed the order publicly), with the remaining 600,000 chips rented through Google Cloud Platform at an estimated $42 billion in remaining performance obligations. Anthropic deploying purchased TPUs in its own facilities — rather than renting all capacity through GCP — signals a shift toward compute ownership, not just procurement. Google has invested $3 billion in Anthropic at approximately 10% equity, with an additional $1 billion committed in January 2025.

To house the direct-purchase TPU clusters, Anthropic selected Fluidstack — a UK-based data center operator — to build custom facilities in Texas and New York, part of a broader $50 billion domestic infrastructure commitment announced in late 2025. Fluidstack handles site construction, cabling, burn-in, and acceptance testing; their CEO described the partnership as enabling “rapid delivery of gigawatts of power” for frontier AI development. The sites are coming online throughout 2026, creating approximately 800 permanent jobs and 2,400 construction jobs.

TPUv7 Ironwood offers approximately 20–30% better raw performance than prior generations, and 50% better price-per-token at the system level versus equivalent GPU configurations. Google cites 60–65% lower power consumption per equivalent FP8 throughput versus GPU alternatives, which matters at gigawatt scale.

The realistic cost advantage for TPUs on LLM training workloads is 30–60% versus H100 clusters in comparable configurations. Claims of 4–10x cost improvement are achievable but apply to specific workloads with highly optimized kernels — not general production workloads. TPUs require custom compiler stacks (XLA), and most model teams optimize for CUDA first. The migration complexity is real: TPU software tooling, debugging infrastructure, and workload portability have historically lagged the Nvidia ecosystem by years. Anthropic’s engineering investment to operate across multiple architectures is a genuine cost that partially offsets the hardware savings, and the thesis depends on Anthropic having solved — or continuing to solve — those tooling gaps at scale.

Why Multi-Accelerator Is Structurally Distinct

Anthropic runs across three accelerator families: Google TPUs, AWS Trainium2, and Nvidia GPUs for research and CUDA-dependent workloads. Each platform is assigned to configurations where its price-performance profile is strongest.

Most frontier model companies rely on a single hyperscaler and a single accelerator family. Anthropic’s portfolio approach produces three advantages that are difficult to replicate quickly.

Cost arbitrage is the most obvious: the blended rate across TPUs and Trainium2 versus an all-Nvidia stack is 30–60% cheaper at comparable throughput on optimized workloads. Supply redundancy is the second: a capacity constraint at one vendor does not halt training. Negotiating leverage is the third, and arguably the most durable: when you are simultaneously committing to 1 million TPUs and a gigawatt-scale Trainium cluster, you have a credible alternative in every procurement conversation. Nvidia’s pricing power is real — but only against buyers with no alternative. It is worth noting that the mere existence of Google TPUs as a credible alternative was reportedly sufficient to save OpenAI — which has never deployed TPUs at scale — roughly 30% on its own Nvidia chip negotiations. Anthropic, which actually uses them at production scale, has considerably more leverage.

The total committed compute across Google TPUs and AWS Trainium now exceeds 2 gigawatts of dedicated capacity. On publicly available estimates, Anthropic’s annualized revenue run rate was in the range of $3–5 billion by late 2025, trending higher. The compute cost per token on this architecture is structurally lower than an Nvidia-only stack at similar scale.

OpenAI: Nvidia Dependency and an ASIC That Hasn’t Shipped

OpenAI’s production compute fleet runs on Nvidia GPUs. Published estimates place active GPU deployments at 100,000–200,000 H100/H200 chips, with additional Blackwell (GB200) orders in progress. These figures are not publicly confirmed. What is confirmed: Bloomberg estimated OpenAI’s quarterly Nvidia GPU spend at approximately $1.8 billion (roughly 6% of Nvidia’s total quarterly revenue at the time). At that run rate, Nvidia dependency is not a footnote — it is the primary cost structure.

Training compute spend in 2024 was approximately $3 billion; inference spend was approximately $1.8 billion. GPT-4 inference alone was projected at roughly $2.3 billion in 2024 — approximately 15x the original training cost. This asymmetry is the core inference economics problem: as models scale and usage grows, inference spend dominates, and the cost structure is entirely Nvidia’s.

Sam Altman has stated OpenAI would own “well over 1 million GPUs” by end of 2025, anchored by the $500 billion Stargate infrastructure program. But compute diversity within Stargate is less developed than the headline suggests.

The Broadcom ASIC: Production Starts H2 2026

On October 13, 2025, OpenAI and Broadcom announced co-development of custom AI accelerators targeting 10 gigawatts of capacity. OpenAI designs the architecture; Broadcom manufactures through TSMC. Production begins H2 2026. Meaningful deployment at scale is a 2027 event at the earliest.

For competitive purposes, this chip does not exist yet. OpenAI’s inference economics through at least 2026 are Nvidia economics. OpenAI also signed a $10 billion deal with Cerebras in January 2026 for 750 megawatts of low-latency inference compute through 2028 — more immediately actionable, with Q1 2026 first deployments, but narrow in scope.

The Stalled Nvidia Mega-Deal

In September 2024, Nvidia and OpenAI announced a letter of intent for a partnership in which Nvidia would invest up to $100 billion in OpenAI as OpenAI deployed at least 10 gigawatts of Nvidia systems.

As of March 2026, no definitive agreement has been signed. Nvidia CFO Colette Kress stated in December 2025 — two months after the announcement — that the $100 billion investment was “never a commitment.” The Wall Street Journal reported negotiations went on ice after internal Nvidia skepticism about OpenAI’s business model. No money has changed hands. Jensen also framed the investment as an invitation to invest from OpenAI, showing OpenAI is essentially at the mercy of Nvidia:

The circular economics were always awkward: Nvidia investing in OpenAI while OpenAI is its largest customer. The stall suggests the announcement was more strategic signaling than procurement commitment.

Microsoft: Announced, Not Shipped

Microsoft’s Maia chip program is the clearest example of the gap between custom silicon ambition and deployment reality.

Maia 100, announced November 2023, was deployed almost exclusively for internal Microsoft staff productivity workloads. It was not designed for generative AI inference at production scale. Two years post-announcement, it has not powered any significant external production service. Microsoft spent $31 billion on Nvidia AI chips in 2024 alone — the most of any cloud provider — while Maia 100 sat on the bench.

Maia 200, announced January 26, 2026, is a more serious design: TSMC 3nm, competitive FP8 throughput, 216GB HBM3e. On paper it is capable. In practice it entered limited deployment in two data centers in early 2026, over two years behind the original roadmap. Delays were caused by last-minute design changes, simulation instability, and high employee turnover affecting chip teams. By the time Maia 200 reaches meaningful Azure deployment, Nvidia’s Blackwell generation will be well-established in production across the industry.

The most telling data point is not the delay itself — chip design is genuinely hard, and two-year slips are not unusual. It is that Microsoft committed $31 billion in a single year to the vendor it was trying to reduce dependence on, while the alternative produced nothing deployable. Custom silicon timelines are measured in years, not quarters. Microsoft is not going to be chip-independent within this model generation cycle.

The Economics at Scale

At committed rates across a gigawatt-scale cluster, the blended cost advantage of TPU + Trainium over all-Nvidia runs roughly $1–2 billion per month on comparable throughput. Annualized, that is $12–$24 billion in structural cost advantage at full 2-gigawatt deployment — before accounting for power efficiency differentials, which are material at that scale. This assumes high utilization and workload fit; realized savings depend on kernel maturity and scheduling efficiency, both of which require sustained engineering investment to sustain.

Inference is projected to account for two-thirds of all compute spend in 2026, up from one-third in 2023. Every token served is a cost event. The cost per token is the unit economics of every API-facing AI business. Anthropic’s blended architecture makes that unit economics structurally better than a lab running equivalent inference on H100s — not by an order of magnitude, but by enough to matter at scale.

Implications

For Anthropic: The compute architecture is not just a cost advantage — it is a capacity advantage. At 2+ gigawatts of committed compute, Anthropic can run training runs that competitors cannot finance at equivalent cost. That compounds model iteration speed. The critical risk is engineering complexity: maintaining efficient workflows across three accelerator families and two compiler stacks requires real investment and creates coordination overhead that Nvidia-only stacks do not.

For OpenAI: The Broadcom ASIC is a credible long-term move, but it does not affect 2026 economics. As H100 spot prices have collapsed, OpenAI’s inference costs have improved — not because of anything OpenAI did, but because oversupply hit the GPU commodity market. That dynamic will tighten again as Blackwell deployment absorbs capacity through 2026. The one scenario that meaningfully narrows the gap: if Blackwell (GB200) delivers on its performance claims, inference cost per token on Nvidia hardware could fall 40–60% from H100 baselines, which compresses Anthropic’s current advantage. That is a real variable to watch.

For Microsoft: Maia 200 arriving two years late is a failure of execution, not strategy. The strategy was correct. But the more important signal for investors is the gap between ambition and output: $31 billion to Nvidia in one year while the internal program produced nothing deployable.

For investors: Compute cost structure is not typically a line item in AI company analysis. It should be. The cost per token served is the unit economics of every API-facing AI business. A 30–60% structural advantage at the compute layer does not translate 1:1 into margin — energy, networking, and engineering overhead all factor in — but it translates substantially. Anthropic’s path to positive gross margin is structurally easier from this compute foundation than from an all-Nvidia stack. At the scale these companies are targeting, the difference is not rounding error.

For the silicon market: The Anthropic model — dual-hyperscaler, multi-accelerator, committed long-term capacity — is likely to become the template for any frontier lab that reaches sufficient scale to negotiate it. Google and Amazon have strong incentives to subsidize Anthropic’s compute to validate their custom silicon against Nvidia at production scale; Anthropic gets below-market compute and Google and AWS get a credible reference proving TPU and Trainium can train state-of-the-art models. Ownership of chips is hard. Integration into the right silicon program is tractable, scalable, and — as this case shows — faster to gigawatt scale than any internal program has managed.

Summary

Anthropic has built the most cost-efficient and diversified compute architecture among frontier AI labs, anchored by Project Rainier — among the largest operational AI compute clusters in the world — and a commitment to 1 million TPUv7 Ironwood chips at 1+ gigawatt scale via a $52 billion deal structured across direct Broadcom purchase and GCP rental. Fluidstack is building the US data centers to house that compute. The blended economics are 30–60% cheaper per token than equivalent Nvidia H100 configurations on optimized workloads.

OpenAI remains structurally dependent on Nvidia through at least 2026. Its Broadcom ASIC is real but does not reach scale until 2027. The stalled $100 billion Nvidia investment deal is PR, not procurement strategy.

Microsoft’s Maia program is two years behind schedule. The company spent $31 billion on Nvidia chips in 2024 while its internal silicon produced nothing deployable for production.

Compute advantage amplifies model advantage. It does not replace it. But in an industry where inference spend is scaling faster than training spend, and where the cost of serving tokens will compound at every major model release, the lab that built the most efficient compute architecture 18 months ago is running a structurally different race than the one that didn’t.

We analyzed public commentary from leading AI operators and hyperscalers and mapped the bottlenecks they consistently cite across the stack.

These constraints arise from a tightly coupled stack:

Each layer has independent lead times and independent bottlenecks. When constraints stack, supply elasticity collapses. Even if GPU output doubles, effective compute-hours can remain scarce.

And that scarcity is not evenly distributed.

The Fiber Analogy Is Incomplete

There are clear parallels to the late-1990s fiber buildout:

• ~$500B in telecom infrastructure investment

• 39 million miles of fiber laid

• <5% utilization at the 2001 peak

• A decade to absorb excess capacity

But there are two critical differences:

1. Fiber could sit idle cheaply.

2. The AI buildout is funded by hyperscaler balance sheets, not junk debt.

Source: Coatue chart of the day

Fiber could sit idle at near-zero marginal cost.

AI infrastructure burns cash every day it sits unused.

• Power contracts

• Cooling costs

• Staffing

• Depreciating GPUs

Compute depreciates in ~5-8 years. Fiber depreciated over decades.

Which brings us to the most important constraint in the stack:

Power.

Power is the binding constraint in AI

Unlike GPUs:

• Power is location-bound.

• Interconnection queues take years.

• Transmission upgrades take longer.

• Generation requires long-term offtake.

• Permitting is political and regional.

You cannot spin up gigawatts the way you spin up GPU clusters. A GPU order cycle is measured in quarters. A transmission upgrade is measured in years.

In many U.S. regions, power availability is already the binding constraint on new AI campuses. Each major data center operator is rushing to lock up as much cheap power as they can.

Source: Public estimates

Last year we projected AI data centers would reach:

• 10% of U.S. grid demand

• ~145 GW by 2028

New bank estimates suggest we hit those numbers before 2027 .

That is not a cyclical constraint.

That is structural.

As an example, CoreWeave’s revenue backlog is scaling faster than its contracted power footprint. Commitments can scale in months. Power cannot.

Power Is Not a Commodity — It’s a Heterogeneous Stack

Deliverable megawatts — firm, high-density, grid-connected — are the scarce asset.

Electricity differs by:

• Baseload vs peaking

• Dispatchability

• Intermittency

• Transmission congestion

• Regulatory structure

• Geographic deliverability

A firm nuclear-backed megawatt co-located with transmission is not equivalent to intermittent solar behind congestion.

As AI campuses move toward multi-gigawatt scale, these differences become economically decisive.

The U.S. grid is not one market. It is ~1,700 fragmented micro-markets with political overlays.

For hyperscalers, that means:

• Tariff fragmentation

• Interconnection queue risk

• Congestion pricing exposure

• Regional political constraints

Further, there are geographical nuances. Texas has more new GW of data centers than all other states combined. Equity markets are re-rating toward regions with power headroom. Texas-heavy utilities (NRG Energy, Constellation Energy) and landholders (Texas Pacific Land) have become indirect AI beneficiaries. Access to power capacity, where it is, and what type of power is a key determinant of which energy players will thrive.

Source: Coatue chart of the day

The Inference Shift Changes Everything

Training is episodic CAPEX. Inference is ongoing load.

Inference-heavy facilities are provisioned with headroom for:

• Peak traffic events

• Failover redundancy

• Geographic routing

• Grid instability

• Future expansion

But utilization fluctuates:

• Model efficiency gains

• Traffic routing shifts

• Customer concentration

• Product cycles

The result: unused firm capacity.

This is the power analog to idle reserved GPU instances.

We’ve already seen compute marketplaces emerge to manage overbuild risk.

Power is next.

Power Is Becoming Financialized

Long-duration power contracts are starting to resemble financial instruments.

We are beginning to see:

• Subleasing of contracted power blocks

• Transfer of interconnection rights

• Monetization of demand flexibility

• Bundled “power + compute” offerings

The next logical step is standardized contracts, clearinghouses, and secondary liquidity.

This is where structural opportunity emerges.

The Strategic Example: Vertical Integration

Alphabet recently paid ~$5B to acquire Intersect Power (founded 2016), a large-scale clean energy developer.

Intersect develops and operates:

• Utility-scale solar

• Battery storage

• Hybrid generation

The acquisition vertically integrates generation with AI infrastructure — reducing basis risk, securing long-term supply, and internalizing power strategy.

This is not an energy investment. It is a compute hedge.

Power is no longer a utility expense. It is a competitive input.

The Venture Opportunity: The “Prime Broker” of Power

The market structure around AI power procurement is immature.

The bottleneck is measurable.

The durable venture layer is not steel and concrete. It is:

• Transaction rails

• Portfolio optimization software

• Structured finance

• Liquidity layers

What does not exist yet:

• A robust secondary clearinghouse for:

  • Capacity blocks

  • Interconnection rights

  • Long-duration contracted megawatts

If inference overbuild materializes, secondary liquidity becomes mandatory.

Why Power Scarcity Persists Even in Overbuild Scenarios

Here’s the key insight:

Even if GPUs overshoot demand by 5–10 years…

Power does not.

Power is:

• Regulated

• Capital intensive

• Politically constrained

• Slow to expand

Memory is cyclical.

Compute pricing compresses.

Power availability remains sticky.

Scarcity shifts pricing power upstream — into the physical layer.

The Structural Feedback Loop

AI lowers the cost of producing software and services.

Lower costs → more adoption.

More adoption → more inference.

More inference → more infrastructure.

More infrastructure → more power demand.

And the more this loop accelerates, the more valuable deliverable megawatts become.

What Happens If We Overbuild?

The central risk is not that AI demand disappears.

It is that parts of the stack overshoot by years.

But here’s the asymmetry:

• Idle GPUs can be redeployed.

• Idle fiber sits cheaply.

• Idle power commitments are expensive.

Compute overbuild compresses margins.

Power overbuild creates optionality.

Those are different regimes.

The Clean Thesis

Power is the lowest-variance, longest-duration constraint in the AI compute stack.

It is:

• Slow moving

• Structurally regulated

• Fragmented

• Financializing

• Becoming programmable

And unlike models or agents, it does not get commoditized by algorithmic improvement.

For venture investors obsessed with model layers, the more durable wedge may sit beneath them:

The transaction, orchestration, and financial infrastructure of energy.

Because wherever portfolios form, software follows.

AI infrastructure is becoming a power portfolio management problem.

And wherever portfolios exist — financial infrastructure follows.

In the early, pre-1990s days of the semiconductor industry, chip designers faced a structural dilemma.They had the technology to specify logic and build powerful custom integrated circuits, but without scalable simulation and verification, their work was more art than infrastructure. Progress was fragile and expensive, and the full system only “proved itself” once it was physically manufactured—meaning bugs were only discovered after fabrication. Eventually, electronic design automation (EDA) changed this by shifting correctness upstream, translating design into verified, executable systems through software.

A similar dynamic currently exists with today’s AI models. As AI shifts from chat to agents running real workflows, the limiting factor isn’t how smart the model is. The limiting factor, like with early chip design, is in whether we can reliably verify the work produced by AI models. Models can already write, browse, reason, and plan at a level sufficient for many professional tasks. But given the absence of reliable reward signals for the models—clear, consistent definitions of success across long-horizon workflows involving tools, judgment, policy, and taste—durable automation remains out of reach.

Enter reinforcement learning. Today, RL environments are playing the same role for AI agents that EDA played for silicon.They translate human intent into executable behavior by making success measurable at scale. But unlike EDA, they must also address how the “spec” for human labor is not deterministic. “Correct” is a moving target with many dimensions, and as agents improve, it becomes harder to measure success with a clean, dependable signal. As a result, RL environments must be living systems that evolve to resist reward hacking and encode judgment. This makes verification—not models—the true bottleneck to automation.

To be clear, EDA and RL aren’t exact analogs. EDA verifies outputs and can sometimes prove correctness; RL verifies incentives and, at best, bounds failure probabilistically. Still, the comparison is useful.A more recent analogy is perhaps the rise of Scale AI in the mid-2010s. At the time, most enterprises lacked in-house machine learning (ML) expertise, and cutting-edge models were still academically constrained. Early buyers wanted data for narrow, static tasks—bounding boxes, transcription, basic classification—primarily for “hard tech” domains, like autonomous driving. Scale’s core insight was that once models became capable enough, progress would be bottlenecked not by algorithms, but by data—and that a sophisticated buyer class would eventually emerge to demand it.

That inflection arrived with the transformer era (2018–2020). As foundation models like GPT-3 demonstrated surprising general capabilities, the constraint shifted again: from data volume to data complexity. The most advanced buyers—AI labs like OpenAI, DeepMind, and Anthropic—suddenly needed feedback over long horizons, as models were now tasked with reasoning, tool use, policy compliance, and multi-step workflows. One-off annotation no longer sufficed.

Over the past three years, this shift has accelerated. Models now need less raw data and far more experience—the kind that encodes enterprise workflows typically locked inside experts’ heads and the rise of expert data. Once data became feedback, labeling became verifiable work.

RL environments are the natural consequence of this transition. They transform workflows into simulated worlds where actions are observable, outcomes can be graded, and learning compounds over time. Environments and evaluations become the new datasets, and training becomes continuous rather than episodic. Replication training is to RL what internet-scale text was to LLMs: the mechanism by which scale produces generalizable infrastructure.

This reframes the competitive landscape. The critical question is no longer, “Who has the best model?” Rather, it’s “Who controls the infrastructure that determines what work is learnable at all?”

Verification Is the Hard Problem—and Solving It Requires Research Depth

Verification is where most RL efforts fail. Defining what “correct” means when there are multiple good answers, partial progress, or policy constraints is fundamentally a research problem, not a tooling problem.

Modern RL training pipelines, therefore, are no longer linear. Models critique humans, humans audit models, and algorithms arbitrate disagreement. The training signal emerges from the system, not any single actor.

As models improve, verification gets harder, not easier. Benchmarks saturate. Reward hacking appears. Graders drift. Environments must be refreshed continuously to stay training-relevant. This demands deep expertise in reward modeling, distribution matching, telemetry, and failure analysis.

As a result, the strongest RL-environment teams look less like SaaS companies and more like applied research organizations. They pair staff-level ML researchers, systems-oriented engineers, and forward-deployed builders with strong domain intuition. Code is rarely the constraint; judgment, data quality, and research velocity are.

Why Coding and Computer Use Are the First RL Markets to Scale

The earliest large-scale adoption of RL environments is happening in coding and computer use. These domains combine three properties that make them ideal early markets:

1. High economic value

2. Dense, machine-observable interaction traces

3. Relatively clean verification loops

Coding already has compilers, tests, and deterministic failure modes. Computer-use tasks—browsers, spreadsheets, CRMs, internal tools—produce replayable state transitions that can be graded and compared. This makes them well-suited for replication training of environments, where agents repeatedly execute long sequences until performance improves.

As a result, coding and computer-use environments are becoming the proving ground for agentic RL. Not because they are the endgame, but because they are the first domains where environment realism, reward design, and eval-to-production correlation can be demonstrated convincingly.

In practice, the deepest environments today are coding-based, such as terminal- and DevOps-centric environments. Coding is programmable because state, tools, and outcomes are explicit and replayable.

These markets matter because they establish trust: with research teams, with buyers, and with budgets. They are the wedge—not the destination.

Who is buying RL environments today?

The early buyers of RL environments look strikingly similar to the first buyers of autonomous-vehicle (AV) data. Just as Scale AI’s early revenue was overwhelmingly concentrated among a handful of AV leaders—Apple, Uber, Lyft, Tesla, and Google/Waymo—frontier AI labs are the dominant initial customers for RL environment companies. This is not a coincidence. RL environments sit upstream of capability gains, and the labs are the only organizations today with both the urgency and the budgets to pay for training infrastructure before it is fully productized. As with AV data in 2018, buyer concentration is high, contracts are large, and vendor selection is tightly coupled to research impact rather than procurement formalities.

Among labs, Anthropic appears to be the single largest buyer across both coding and computer-use environments. Across vendors focused on coding and computer use, Anthropic is estimated to be spending *on the order of* tens of millions annually on RL environments, making it the clearest anchor customer in the market today. Anthropic signed these contracts in 2025 and are already scaling rapidly; based on current trajectories, their aggregate lab spend on RL environments is likely to grow 3–5× into 2026 as environments move from experimentation to a core part of model training. OpenAI’s total spend is harder to isolate due to its broader data footprint, but it has already signed multiple seven-figure RL-environment and human-data contracts, indicating similar strategic intent.

Labs are also buying adjacent inputs aggressively, particularly human expertise for coding and long-horizon reasoning.

These purchases reinforce a critical point: labs are not just buying models or raw data—they are buying training throughput, and RL environments are becoming a first-class line item in that budget.

Enterprises, by contrast, are entering through a different door. Rather than buying standalone environments, they are purchasing forward-deployed, outcome-oriented solutions that embed RL environments under the hood. This is the story that Applied Compute is telling and why investors are so excited about its trajectory.

Product-led companies like Glean, Notion, and Cursor are running pilots or building in-house environments while evaluating vendors. This pattern mirrors earlier infrastructure cycles: labs fund the tooling when it is raw and research-heavy; enterprises adopt once it is wrapped around concrete workflows and measurable ROI.

This is why it is critical to have lab success in the early days to achieve escape velocity.

The implication is clear. RL environment companies will be built on lab money first, just as Scale was built on AV customers before expanding into enterprise ML and later LLM/AI research. Understanding which vendors are winning lab mindshare today is therefore the clearest leading indicator of who will control the training layer tomorrow.

The Real Prize: Long-Form White-Collar Work Across Many Environments

The long-term opportunity for RL environments lies in long-form workflows that span multiple tools, not single-step tasks. These workflows are how value gets created across finance, operations, go-to-market, and engineering.

A finance workflow might move from spreadsheet analysis to browser-based research, into an ERP system, and back into documents for approval and audit. An insurance claim touches intake portals, policy systems, historical databases, email, and compliance tooling. Legal work spans research databases, drafting tools, revision cycles, and risk review. In each case, success is judged by downstream outcomes—risk reduction, correctness under audit, or decision quality—not by any single action.

The same pattern holds across core operating roles. Sales moves across CRMs, email, calendars, pricing tools, and pipeline forecasting. Marketing spans analytics dashboards, ad platforms, creative tools, and experimentation systems. Customer support blends ticketing systems, product logs, documentation, and multi-turn communication. Developer workflows shift between IDEs, browsers, repositories, test runners, and CI systems. These are composed systems of environments, not isolated tasks.

In other words, these are not single environments—they are composed systems of environments. Training agents for them requires:

• Persistent state across tools

• Cross-environment credit assignment

• Verification tolerant of multiple valid paths

• Rewards tied to business outcomes, not toy metrics

This is where most early systems will break. One-off environments saturate quickly, and narrow benchmarks stop teaching. The future belongs to teams that can build environment factories: systems that assemble, orchestrate, and refresh multi-environment workflows continuously.

In this regime, progress comes from repetition at scale. Replication training—running the same workflow thousands of times across slightly varied environments—is how learning compounds. This is an emerging field in RL environments and will determine the market leader over time. The bottleneck is no longer model capacity. It is environment fidelity, verification quality, and orchestration scale.

The dynamics are shifting from building models to earning trust, access, and validation inside the frontier labs.

What does the winning RL Environment company look like?

Between 2026 and 2030, the RL environment market will narrow decisively. What today looks like roughly 20 seed- to Series A-stage companies on relatively similar footing—be they forward-deployed teams, early environment builders, or research-heavy startups—will resolve into three to five market leaders, with one to two dominant platforms pulling meaningfully ahead.

This is not a land-grab driven by environment count or early demos. It is a selection process driven by two reinforcing advantages:

1. Who earns “thought partner” trust from the frontier AI labs early, and

2. Who builds the research organization capable of industrializing replication training and verification.

Labs today are capturing low-hanging fruit in many simple environments to teach AI to use tools, especially to train v0 of computer use. In the future, labs will optimize less for the volume of environments shipped, as basic applications are trained. They will optimize for complexity, quality, and what comes next, and they will increasingly concentrate their spend on the few teams that help them push the frontier forward.

Here are six key principles of what will determine who breaks out in 2026-2030:

1. Frontier labs will choose embedded thought partners, not on-demand vendors

2. RL environments will evolve from brittle artifacts to automated infrastructure

3. Replication/environment training and verification define the research moat

4. Lab trust and research depth are self-reinforcing

5. Frontier lab work is the foundation; enterprise work is the multiplier

6. Depth in core and complex domains (i.e., coding) beats environment breadth (lots of environment apps)

1. Frontier Labs Will Choose Embedded Thought Partners, Not On-Demand Vendors

RL environments are currently still immature. Many are single-application, brittle to UI or workflow changes, and partly manual in how tasks are defined or graded. This is acceptable today, but it will not be as agent autonomy increases.

Agents can already run autonomously for two to three hours so long as the setting is constrained. As autonomy increases, training shifts from isolated tasks to long-horizon workflows spanning multiple environments, where state persists, decisions compound, and success only comes much later. This shift fundamentally changes what labs need.

As a result, frontier labs aren’t looking for vendors who can “build environments on request.” They’re looking for thought partners: teams who can help define how RL infrastructure must evolve as agents become more autonomous. In practice, this means co-designing new kinds of environments, stress-testing verification in ambiguous settings, and experimenting with replication training regimes that don’t yet have established playbooks.

Buyer concentration is a feature, not a bug. Winning a few anchor lab relationships matters far more than broad distribution. As a reminder, Scale AI had over 80% revenue concentration in roughly five AV customers going into 2019-2020.

‍Key Takeaway: Labs gravitate toward teams they trust, like working with, and treat as extensions of their own research organizations. Frontier labs are increasingly the gravitational center for RL talent. Startups that maintain live research pipelines into academia and labs inherit that gravity; others slowly fall out of relevance.

‍2. Environments Become Environment Factories: From Brittle Tasks to Automated Infrastructure

The technical evolution of RL environments is clear. RL environments will move from single-app, hand-curated tasks; annual or semi-manual grading; and fragile assumptions about tools and interfaces, toward multi-environment workflows; automated task generation and variation, and hybrid verification systems that scale with agent capability.

The end state isn’t “better environments.” It’s automated RL infrastructure: environment factories that continuously assemble, orchestrate, test, and refresh environments as agents improve and software ecosystems evolve.

This shift will be gradual. RL environments will stay largely manual for some time; if the entire pipeline were already automatable, frontier labs would simply build it in-house. The fact that they haven’t is precisely why this market exists.

Key Takeaway: As agents improve, environment quality—not environment count—becomes the binding constraint. Labs will reward teams that can absorb complexity, keep environments stable, and anticipate what the next generation of training will require.

3. Replication training and hybrid verification are the compounding technical moat

The core research challenge over the next five years is not simply building more RL environments, but turning environment training into a scalable system. This is where replication training becomes the defining primitive.

Replication training replaces bespoke, one-off environment building with controlled repetition at scale. Instead of constantly inventing new tasks, teams pick a small set of high-value, long-form workflows and run them thousands of times across slightly varied environments—different inputs, starting states, tool access, and constraints—while keeping the underlying workflow stable. Learning compounds not from novelty, but from repeated exposure across changing conditions.

Building reliable, extensible replication training is non-trivial. Replication training requires:

• Deterministic reset and replay, so workflows can be run repeatedly without drift

• Environment abstraction layers, which allows for variation without breaking realism

• Parallel orchestration, to run large numbers of long-horizon trajectories

• Instrumentation and telemetry, to surface where and why agents fail over time

The objective is to extract maximal learning signal from a minimal set of environments—driving down marginal cost while preserving fidelity.

Verification is the harder half of the system. In long-form, multi-environment workflows, there’s rarely a single correct answer, and as agents improve, the reward signals often become noisier, not cleaner. That’s why winning systems use closed, hybrid verification loops: experts score model outputs, models review expert judgments, humans supervise and correct models, and algorithms reconcile disagreement, detect drift, and curate edge cases

In this regime, data work becomes intellectual—the job is to construct and maintain a reliable training signal in the presence of ambiguity. Replication training only works if this verification loop remains stable as scale increases. That stability is a research problem, not a tooling problem.

Prompt optimization systems like GEPA are not alternatives to RL infrastructure, but adjacent primitives that plug into the same replication and verification systems.

The core skills required to build this system are rare and cross-disciplinary:

1. Systems engineering, to manage orchestration, replay, and scale

2. Applied ML research, to design rewards and diagnose failure modes

3. Human-in-the-loop design, to balance automation and judgment

4. Product intuition, to decide which variations actually teach the model something new

Teams that master these skills turn replication training into a compounding advantage—and define the real moat in RL environments.

Key Takeaway: Replication training only becomes a moat when it’s paired with a stable, closed-loop hybrid verification system—because the winner isn’t the team with the most environments, it’s the team that can reliably extract learning signal from a small set of long-form workflows at scale. That combo (repeatable training + compounding, ambiguity-resistant verification) drives down marginal cost, prevents drift, and turns every run into durable advantage.

4. Research Lineage Compounds: Trust → Talent → Frontier Exposure → Better Systems

These dynamics create a powerful flywheel, and research credibility is the entry ticket. Teams with genuine research lineage—often signaled simply by who built them—earn trust faster with both labs and investors.

Key Takeaway: Teams with strong research organizations win deeper lab engagements. Those engagements expose them to frontier failure modes earlier than the rest of the market. That exposure feeds directly into better replication training systems, stronger verification pipelines, and lower marginal cost per new environment. Over time, this compounds into durable technical advantage.

5. Lab-first builds the core; Enterprise scales the distribution

While lab trust, research depth, and talent reinforce one another, they are not equal inputs. If one must serve as the foundation for breakout success, it is unequivocally frontier lab work. This is where RL infrastructure is actually built. Enterprise work, by contrast, is where that infrastructure is applied, customized, and monetized.

Frontier labs force teams to solve the hardest problems first. Environments must generalize across models. Replication training must work at scale. Verification must hold up under long-horizon ambiguity. Orchestration, replay, and evaluation-to-performance correlation cannot rely on customer-specific shortcuts. Lab buyers are purchasing infrastructure primitives—environments, tasks, training loops—not bespoke outcomes. That pressure produces reusable systems rather than one-off solutions.

Enterprise demand today looks different. Most enterprises are not buying RL environments or training infrastructure directly. They are buying agentic applications tied to concrete workflows and KPIs, often delivered through forward-deployed engineering. This work is valuable, but inherently more custom. Enterprise agents benefit disproportionately from infrastructure that has already been hardened upstream in lab settings—effectively “RL menus” rather than bespoke training from scratch.

The asymmetry matters. Infrastructure built for labs lowers the marginal cost, risk, and time-to-value of enterprise deployments. Replication training pipelines, environment factories, and hybrid verification systems developed for lab use increasingly power enterprise work behind the scenes. Over time, what looks like bespoke enterprise delivery becomes configuration on top of lab-derived infrastructure.

Key Takeaway: Frontier lab work is the foundation. It produces the training and verification infrastructure that later makes enterprise deployments cheaper, faster, and more scalable. The teams that break out will be lab-first platforms that use enterprise demand as a multiplier—not the other way around.

6. Depth in Core, Complex Domains Beats Environment Breadth

In the early stages of building enduring RL infrastructure, depth in a small number of complex domains matters far more than breadth across many shallow ones. Not all environments are equally valuable. The environments that matter most are those where success is hard to define, trajectories are long, tools are central, and failure modes are subtle. Coding and computer use sit squarely in this category, with coding as the holy grail.

Coding is not just another task—it is a meta-domain. It combines long-horizon reasoning, tool invocation, stateful context, error recovery, and verifiable outcomes. Crucially, it also offers dense feedback: tests pass or fail, programs compile or break, diffs can be evaluated, and performance can be measured. This makes coding one of the few domains where high-quality reinforcement signals are available at scale, even as the task itself remains cognitively demanding.

This is not just a theoretical advantage; it is already reflected in market demand. Coding is the largest application vertical for the leading AI labs. Claude Code reached roughly $1 billion in ARR within six months. Microsoft Copilot is already a multi-billion-dollar business, and OpenAI’s Codex is generating hundreds of millions in annualized revenue. These products sit at the frontier of long-horizon, tool-heavy agent behavior and are natural early consumers of RL environments.

The most successful AI-native applications—and the likely first enterprise adopters of RL environments to stay competitive—are overwhelmingly coding-centric. Cursor is approaching $2B in ARR, with teams like Windsurf/Cognition close behind, and a fast-growing cohort of newer startups such as Cline and Kilo AI emerging. These companies are not experimenting at the edges; they are pushing agents into production workflows where reliability, verification, and continuous improvement matter.

By contrast, many RL environment companies are adopting environment-first strategies optimizing for surface area rather than depth. Teams build dozens of narrow environments—CRM editing, scheduling / email drafting, Slack clones—that demonstrate capability but rarely compound. These environments are short-horizon, weakly coupled to tools, and often rely on brittle heuristics or human-in-the-loop grading. They are easy to demo, but hard to generalize, and they produce limited reusable infrastructure. They provide initial value to labs training computer use, but will quickly commoditize.

Depth compounds in ways breadth does not. Teams that invest deeply in coding and complex computer-use environments are forced to solve hard, reusable problems early: scalable replication training, hybrid verification, trajectory replay, and eval-to-performance correlation. Improvements in these systems transfer across tasks within the domain and, over time, into adjacent domains like data analysis, DevOps, and general computer use.

This is why coding and computer use are the fastest initial pickup markets for RL environments. They sit at the intersection of high economic value, high task complexity, and strong verifiability. Teams that win here are not just shipping better agents—they are building the core training and evaluation infrastructure that makes broader generalization possible later.

Key Takeaway: Early advantage in RL environments accrues to teams that go deep in a small number of complex, high-signal domains—especially coding and computer use. Breadth can be added later. Depth, once skipped, is hard to recover. Coding is the single best area of expertise to build a foundation off of.

Market Landscape Today in 2026

The emergence of RL environments doesn’t displace incumbent labeling vendors, but it reshapes where value accrues. Traditional labeling is roughly a $5B market today (growing in excess of 50% YoY). The broader AI training data market ahead is likely to be significantly larger—but structurally different. Scale and workforce orchestration mattered in the labeling cycle. In the next one, research depth, domain specialization, and systems-level integration matter far more.

Incumbents like Surge AI, Mercor, Turing, Invisible, and others already generate billions in aggregate revenue serving frontier labs with expert labeling, fine-tuning data, and evaluations. That demand is expanding, not shrinking—especially in coding, which is already the highest-value expert vertical across these platforms. Coding-heavy expert work is where long-horizon reasoning, tool use, and verifiability converge, making it the natural bridge from labeling to RL environments.

That said, most incumbents are still optimized for a services-first, episodic workflow model, not for continuous learning systems. They aren’t research-first orgs at their core today. RL environments require reusable environment abstractions, deterministic task design, scalable replication training, and tight eval-to-performance feedback loops. These are infrastructure and research problems, not labor problems. The founding makeup of teams like Bespoke and Applied Compute are a mix of ML Scientists/Engineers and FDEs, not labelers, project managers, and software engineers.

It would not be surprising if one or two incumbents with strong research organizations successfully adapt to this new vertical. Surge AI is the clearest candidate today, and Snorkel has the intellectual foundations even if it lacks scale. But structurally, this market favors specialists built natively for RL environments. Depth compounds faster than breadth, and teams designed from day one to build training systems—not just manage expert throughput—are best positioned to capture the long-term value.

Which companies will win in 2030?

By 2030, the winners in RL environments will not be the teams with the most demos, the widest environment catalogs, or the largest services organizations. They will be the teams that build real infrastructure—and do so in close partnership with frontier labs to industrialize replication training, verification, and long-horizon environment orchestration.

That reality narrows the field. There will likely be 3 to 5 significant winners as labs concentrate spend among a small set of trusted partners. This market may end up resembling data labeling, which already has roughly three $1B+ revenue players (Scale, Mercor, Surge) and more than five $100M+ players (Turing, Micro1, Invisible, Handshake, etc.).

Most early-stage players lack one of the two non-negotiables: deep research capability or sustained lab trust. Without both, teams risk stalling as forward-deployed shops or commoditized environment builders as labs consolidate spend with a small number of thought partners. Incumbent labeling vendors will participate in this market, but few are structurally positioned to lead it.

In the end, this market will not be won by teams that apply RL environments fastest, but by those that help labs define how RL environments should exist at all. The enduring platforms will be built by teams that partner with frontier labs to turn ad-hoc experimentation into replication training systems, fragile graders into robust verification loops, and one-off demos into durable infrastructure.

That work is slow, research-heavy, and often invisible from the outside, but it compounds. And by the time RL environments become an obvious, enterprise-scale category, the winners will already be decided: not by market share today, but by who the labs trusted to build the training layer when it still mattered most.

As AI inference moves from experimentation to production, the infrastructure layer is no longer converging on a single “best” model. Instead, it is bifurcating into two durable and economically attractive markets, each optimized for a different constraint.

On one side are reserved and hourly compute platforms, designed to deliver predictability, control, and determinism for customers willing to operate infrastructure directly.

On the other are inference APIs, which trade some control for scale, absorbing utilization risk and abstracting complexity in exchange for cost efficiency and speed.

Understanding this split—and the tradeoffs it implies around utilization, performance, and deployment—is essential to evaluating where value will accrue as inference becomes ubiquitous.

Reserved Compute Platforms vs Inference APIs: Two Attractive Markets with Different Strengths

The AI infrastructure market is bifurcating into two durable and attractive segments, each optimized for different customer priorities and economic tradeoffs:

• Reserved / hourly compute platforms optimize for predictability and control

• Inference APIs optimize for utilization efficiency and cost abstraction

Neither is categorically superior; each wins in a different region of the workload and buyer space.

1) Reserved / Hourly Compute Platforms: Predictability and Control

Compute platforms offering hourly or reserved GPU instances (e.g., SF Compute, Modal–style offerings) are attractive because they deliver determinism.

What customers are buying

• Guaranteed access to scarce GPUs

• Predictable performance and capacity

• Full control over the runtime and stack

• Clear, stable unit economics (GPU-hours)

Why this is valuable

• Many workloads require consistency more than elasticity

• Teams want to size infrastructure once and operate against it

• Enterprises value control, compliance, and reproducibility

Economic profile

• High utilization per customer when workloads are well sized

• Revenue tied to long-lived contracts or reservations

• Smaller number of customers, but:

  • Larger ACVs

  • Lower churn

  • Clear capacity planning

Tradeoff

• Limited aggregation across customers

• Growth scales linearly with GPU fleet expansion

• TAM constrained by customers willing to operate infrastructure

This model resembles:

Early AWS EC2 reserved instances, CoreWeave-style compute, or traditional hosting—capital intensive, but predictable.

2) Inference APIs: Utilization, Cost Abstraction, and Scale

Inference APIs are attractive for the opposite reason: they absorb complexity and utilization risk on behalf of customers.

What customers are buying

• Tokens or requests, not hardware

• Elastic capacity without provisioning

• Built-in batching, routing, and scaling

• Operational simplicity and fast time-to-production

Why this is valuable

• Most inference workloads are bursty and uneven

• Most teams do not want to manage GPU fleets

• Cost predictability at the application level matters more than raw unit cost

Economic profile

• Lower utilization per customer

• Higher utilization at the platform level via aggregation

• Revenue scales with:

  • Number of customers

  • Number of use cases per customer

  • Overall inference volume

Tradeoff

• Less deterministic per-request performance

• More exposure to price compression at the execution layer

• Requires sophisticated systems to maintain margins

This model resembles:

Stripe, Twilio, or Datadog—abstracting infrastructure into a product.

3) The Core Tradeoff: Predictability vs Aggregation

Key insight

Reserved compute platforms win by making one customer predictable.

Inference APIs win by making many customers efficient together.

4) Why Both Markets Remain Attractive Long-Term

• AI workloads are diversifying, not converging

• Some buyers will always prioritize control and determinism

• Others will always prioritize speed, elasticity, and simplicity

As a result:

• Reserved compute captures value where predictability matters

• Inference APIs capture value where utilization and abstraction matter

Growth in one does not cannibalize the other; it often expands the total market.

Clean takeaway

Reserved compute platforms and inference APIs are both attractive, but for opposite reasons: one monetizes predictability with fewer, larger customers; the other monetizes utilization through aggregation across many smaller customers.

Inference Optimization

Utilization vs. Systems-Level Optimization: Why Utilization Dominates Inference Economics

While numerous systems-level optimization companies have developed techniques to enhance performance, only inference APIs—which directly manage GPU resources—possess the capability to optimize utilization.

A common intuition in inference infrastructure is that systems-level optimizations—higher tokens/sec, better kernels, smarter schedulers—are the primary drivers of durable advantage. In practice, for inference APIs, utilization is the first-order determinant of cost, margin, and competitiveness, while systems-level optimizations are second-order and increasingly commoditized.

1) The Two Levers Are Fundamentally Asymmetric

There are two ways to lower effective cost per token:

1. Increase tokens per second (performance)

2. Increase utilization (time GPUs are busy)

These levers behave very differently.

• Performance improvements reduce cost only if demand exists to consume the additional throughput.

• Utilization improvements reduce cost unconditionally, by eliminating idle time.

For most inference workloads—especially user-facing, bursty workloads—idle time dominates waste, not insufficient throughput.

2) Performance Advantages Compress Because Techniques Are Open and Diffuse

(With Representative Vendors)

Across modern inference APIs running similar model classes (e.g., 70B-parameter LLMs on A100/H100-class hardware), real-world tokens/sec variance is relatively narrow and shrinking.

Observed production ranges typically look like:

• Lower-throughput, aggregation-optimized APIs:
  ~120–150 tokens/sec

• Higher-throughput, latency-optimized APIs:
  ~220–260 tokens/sec

This ~1.3–2.1× spread reflects different optimization priorities rather than fundamentally different technology.

Representative Vendors by Throughput Orientation

(Illustrative, not exhaustive; varies by model, region, and configuration)

Higher Tokens/sec (Latency / Performance Optimized)

These platforms tend to push hardest on kernel-level and model-specific optimizations, often trading off breadth or flexibility.

• Fireworks AI

  • Heavy focus on latency and throughput (tokens/s)

  • Aggressive batching and kernel tuning

  • Narrower operational envelope per model

• BaseTen (dedicated / reserved setups)

  • High sustained throughput when utilization is high

  • Performance stability prioritized over elasticity

  • Best-case tokens/sec when capacity is fully consumed

• Select hyperscaler inference offerings

  • Deep integration with hardware-specific runtimes

  • Strong performance under controlled workloads

Lower Tokens/sec (Aggregation / Cost Optimized)

These platforms optimize for heterogeneous demand, model breadth, and utilization, rather than peak throughput.

• DeepInfra

  • Broad model catalog

  • Conservative batching to support mixed workloads

  • Optimized for platform-level utilization rather than per-model speed

• Together.ai (non-reserved, general API paths)

  • Trades some peak throughput for flexibility and rapid model availability

  • Performance varies widely by model and load

• General multi-tenant inference APIs

  • Support many models and traffic shapes concurrently

  • Throughput traded off in favor of reliability and aggregation

Why These Differences Persist—but Compress

The key observation is not who is faster, but why the gap is bounded:

• The fastest vendors rely on:

  • Flash-Attention variants

  • TensorRT-LLM

  • Triton kernels

  • KV-cache and batching strategies

• These techniques are:

  • Largely open source

  • Rapidly upstreamed into shared libraries

  • Adopted across vendors within quarters

As a result:

• Performance leaders change frequently

• The slowest credible APIs rarely fall more than ~2× behind

• The economic impact of performance differences is often dominated by utilization

Key Implication

Being the fastest inference API confers temporary advantage; being the most utilized confers durable advantage.

3) Utilization Variance Is Larger and More Durable

By contrast, utilization variance across inference platforms is both larger and more persistent.

It is common to observe:

• ~40–55% utilization on lightly aggregated or partially reserved systems

• ~70–85% utilization on well-designed, multi-tenant inference APIs

This represents a ~1.5–2.0× difference in effective cost, independent of tokens/sec.

Crucially:

• Utilization is driven by aggregation, customer mix, traffic heterogeneity, admission control, and product abstraction.

• These are business-model and architecture choices, not kernel tweaks.

• They do not diffuse via GitHub.

4) Concrete Comparison: Highest vs Lowest Tokens/sec Inference APIs

To make the tradeoff tangible, compare the best-case performance leader with a conservative, aggregation-optimized inference API running the same model class under production conditions.

For a 70B-parameter LLM on A100/H100-class hardware:

• Highest-tokens/sec inference API: ~230–260 tokens/sec

  • Aggressive kernel fusion

  • Highly tuned attention implementations

  • Large batch sizes under load

  • Often prioritizes throughput/latency over flexibility

• Lowest-tokens/sec (credible) inference API: ~120–150 tokens/sec

  • Conservative batching

  • Broader model catalog

  • More heterogeneous workloads

  • Optimized for reliability and aggregation

This is roughly a ~1.7–2.1× gap in raw tokens/sec.

Now layer in utilization:

Despite being nearly 2× slower on paper, the aggregation-optimized platform can deliver meaningfully better economics because its GPUs are busy far more of the time.

Two observations follow:

1. The fastest API is rarely the cheapest in practice. Peak tokens/sec only matters when demand is perfectly aligned and sustained—uncommon outside narrow workloads.

2. The performance gap is bounded and compressing. Even the slowest serious inference APIs are usually within ~2× of the fastest, while utilization differences regularly exceed that.

In inference APIs, the difference between “best” and “worst” performance is bounded; the difference between “busy” and “idle” infrastructure is decisive.

5) When Systems-Level Optimization Still Matters (Second-Order)

This does not imply systems work is unimportant.

Performance optimizations matter most when:

• Utilization is already high and stable

• Latency directly impacts revenue

• Workloads are homogeneous and predictable

• SLAs are strict

In these regimes, performance improvements compound on top of utilization.

However, for most inference APIs, performance is a multiplier on utilization, not a substitute for it.

Clean takeaway

Utilization variance dominates performance variance—and unlike performance optimizations, utilization advantages do not open-source away.

Strategic Synthesis

Utilization determines whether an inference API is competitive at all.

Systems-level optimizations determine how competitive it is once it is.

Because performance techniques are largely open, shared, and fast-diffusing, durable advantage in inference APIs comes from aggregation, abstraction, and control, not raw execution speed.

Utilization Assumptions (for Pricing & Cost Models)

All cost and unit-economics comparisons in this report assume observed production utilization, not theoretical peak throughput. A critical distinction throughout is between customer-level utilization and platform-level utilization—they behave very differently and are often conflated.

Key Definitions

• Customer-level utilization: How busy GPUs are for a single customer or single workload.

• Platform-level utilization: How busy GPUs are across the entire platform, aggregating many customers and workloads.

Most misunderstandings come from applying customer-level intuition to platform-level economics.

Utilization Tiers Used Throughout

10–40% Utilization: Customer-Level Reality (Small, Isolated, or Reserved)

This band is normal for individual customers, even when systems are well engineered.

Who lives here

• Series A–C AI startups
  Enterprise internal AI teams

• Teams using reserved GPUs or BYOC deployments

• Products with one or two core AI features

Why utilization falls in this range

• Capacity is provisioned for p95–p99 traffic, not averages

• Demand is bursty and diurnal
  Multiple models or pipelines fragment GPU memory

• Latency SLOs prioritize headroom over cost efficiency

Representative examples

• A Series B startup running reserved A100s for a customer-facing copilot

• A Fortune 500 company deploying an internal HR or IT chatbot

• A SaaS product with one AI feature used heavily during business hours

Key point

Utilization in the 10–40% range reflects traffic isolation and risk tolerance, not poor execution.

This is the dominant reality for most companies below platform scale.

40–60% Utilization: Mature Single-Tenant or Light Aggregation

This band reflects operational maturity without hyperscale aggregation.

Who lives here

• Mid-size SaaS companies with steady AI demand

• Large enterprises with a single, heavily used internal platform

• Vendors offering semi-dedicated capacity to a small number of customers

Why utilization improves

• Demand becomes more predictable

• Fewer models are active per GPU
  Batching and admission control are consistently applied

Representative examples

• A public AI SaaS company with steady weekday traffic

• A large enterprise analytics copilot used company-wide

Important constraint

• This is often the upper bound for single-tenant or lightly multi-tenant systems

• Pushing beyond ~60% usually requires:

  • Aggregating many customers, or

  • Accepting higher tail latency

60–85% Utilization: Platform-Level Aggregation (Large Platforms)

This band is structurally limited to platforms that aggregate many customers.

Who lives here

• Multi-tenant inference APIs

• Large inference platforms serving many startups

• Hyperscaler-managed inference services

Why utilization is high

• Many small, asynchronous demand curves smooth out

• Peaks rarely align (statistical multiplexing)

• Platforms can batch aggressively and queue intelligently

• Startup customers tolerate shared infrastructure and mild latency variance

Representative examples

• OpenAI-style public APIs

• Large inference platforms serving hundreds or thousands of startups

Critical caveat

• Sustained utilization above ~85% sharply increases tail latency and failure risk

• Most user-facing platforms deliberately operate below this ceiling

Implications for the Rest of This Report

1. Startup-heavy platforms can still achieve high utilization
   Individual startup customers are low-to-moderate utilization, but aggregating many of them drives platform-level utilization into the 60–85% range.

2. Reserved and BYOC models structurally reduce utilization
   Pinning capacity to customers collapses the benefits of aggregation.

3. Utilization differences dominate cost outcomes
   A platform running at 70–80% utilization will typically outperform a faster system running at 40–50%, even if tokens/sec is lower.

Utilization is primarily a function of aggregation and abstraction—not company size or engineering quality.

All subsequent cost, performance, and market analysis should be interpreted through this lens.

Why 10–25% Utilization Is Real (and Common)

When people hear “10–25% GPU utilization,” they often imagine mismanagement. In practice, it usually reflects normal product behavior, not incompetence.

1) Early-Stage and Growth-Stage Products

For most inference-backed products before real scale:

• Traffic is bursty (peaks during business hours, troughs at night)

• Feature usage is uneven (one agent or endpoint dominates)

• Capacity is sized for p95 or p99, not average

Result:

Even “healthy” systems sit at 10–30% average utilization.

This mirrors:

• Web servers pre-autoscaling

• Databases sized for peak QPS

• Early SaaS infra economics in general

2) Reserved Capacity Is Sized for Worst Case, Not Average

Any system with:

• SLAs

• Latency SLOs

• Human-facing UX

must provision for spikes, not medians.

Example:

• Peak demand: 5× average

• You size for peak

• Average utilization → ~20%

This is by design, not error.

3) Multi-Model and Multi-Tenant Fragmentation

Utilization collapses when:

• Multiple models are deployed in parallel

• Each has independent traffic patterns

• GPU memory constraints prevent consolidation

Even if total demand is high:

• Individual GPUs may be underutilized

• Fragmentation blocks efficient packing

This is especially common in:

• Enterprise copilots

• R&D-heavy teams

• Products experimenting with multiple models

4) Why Serverless APIs See 60–85% While Customers See 10–25%

This is the key contrast.

Single customer / reserved setup

• Demand curve = narrow

• Variance = high

• Utilization = low

Aggregated serverless platform

• Demand curves from thousands of customers overlap

• Variance smooths out

• Utilization rises to 60–85%

This is not magic — it’s statistical multiplexing.

Is 10–25% the Floor or the Average?

Important nuance:

• 10–25% is common for:

  • Early-stage products

  • Reserved GPUs

  • Internal tools

• 40–60% is achievable for:

  • Mature SaaS with predictable usage

  • Well-tuned admission control

• 70–85% is rare outside:

  • Large serverless platforms

  • Highly constrained, always-on workloads

Sustained >85% utilization is dangerous unless you’re:

• Willing to drop requests

• Willing to increase latency

Quantitative Reality Check

Consider a product with:

• 8 hours/day of peak usage

• 16 hours/day of low usage

• 2× peak-to-average ratio

Even with good smoothing:

That’s a well-run system.

Now add:

• Model experimentation

• Geographic skew

• Release cycles

You’re back in the 20–30% range quickly.

Why This Matters Strategically

This is why:

• Reserved GPU inference looks cheap on paper and expensive in reality

• Performance optimizations often disappoint economically in isolation

• API-based inference dominates long-term share

It also explains why:

• Systems that maximize utilization outperform those that maximize peak throughput

• Control planes beat raw execution engines

Inference APIs vs. Customer Utilization: Why Startup-Heavy Platforms Can Run High Utilization

A common misunderstanding in inference economics is the assumption that serving mostly startups implies low utilization. This confusion comes from conflating customer-level utilization with platform-level utilization—two very different things.

Customer-Level vs Platform-Level Utilization

At the customer level (typical startup behavior):

• Utilization is often low (10–30%)

• Traffic is bursty and uneven

• Workloads are small and episodic

• Capacity is sized for peaks, not averages
  

This is true for most startups, and it is entirely rational.

At the platform level (inference APIs):

• Hundreds or thousands of independent demand curves are aggregated

• Peak usage times rarely align

• Idle capacity smooths out statistically

This is classic statistical multiplexing.

A platform serving many low-utilization customers can itself operate at 60–85% utilization.

This is not unique to inference. It is exactly how infrastructure platforms like:

• AWS EC2

• Datadog

• Stripe

• Twilio

achieved strong unit economics early—despite serving predominantly small customers.

Why Startup Customers Often Improve Platform Utilization

Counterintuitively, startup-heavy customer bases are often better for utilization than a few large customers.

1) Asynchronous, Heterogeneous Workloads

Startups tend to:

• Operate across time zones

• Have different peak hours

• Use different models, endpoints, and request shapes

This creates natural demand smoothing.

Contrast this with:

• One large enterprise customer

• One internal copilot

• One dominant model

Those scenarios concentrate demand and make it harder to keep GPUs busy.

2) Fewer Hard SLAs

Startup customers are typically more tolerant of:

• Mild latency variance

• Queueing under load

• Shared infrastructure

That tolerance allows inference APIs to:

• Batch more aggressively

• Apply stricter admission control

• Run GPUs closer to saturation

By contrast, enterprise customers often impose:

• Dedicated capacity requirements

• Strict tail-latency guarantees

• Isolation constraints

All of which structurally reduce utilization.

3) Minimal Demand for Reservation or Isolation

Most startups:

• Do not request dedicated GPUs

• Do not require VPC isolation

• Default to fully serverless APIs

This allows inference platforms to:

• Pool capacity globally

• Move workloads dynamically

• Optimize scheduling and batching across customers

That flexibility is the core utilization advantage of inference APIs.

When Utilization Drops for Inference Platforms

There are two scenarios where even inference APIs experience utilization pressure.

1) Growth in Reserved or Dedicated Capacity

If a platform:

• Pins GPUs to specific customers

• Sells large amounts of reserved capacity

• Guarantees isolation or fixed latency

Then utilization begins to resemble:

• 40–60% in well-sized cases

• 20–40% when demand variance is high

This is why reserved inference tends to be expansion revenue, not the economic core.

2) Heavy Skew to a Single Model or Workload

Utilization also drops when:

• A single model dominates demand

• Requests cluster in time

• Latency SLOs are strict

In these cases:

• Capacity must be overprovisioned

• Scheduling flexibility decreases

However, these are also situations where:

• Pricing power increases, or

• Routing and scheduling logic becomes strategically valuable

Where Inference APIs (i.e. DeepInfra, Fireworks, Together) Sit (Illustrative, Not Unique)

For a modern inference API with:

• A serverless model

• A startup-heavy customer base

• A broad catalog of models

It is reasonable to expect:

• Platform-level utilization: ~60–80%

• Customer-level utilization: often 10–25%

• Effective economics: closer to hyperscale APIs than reserved-GPU platforms

In other words:

Startup customers are low-utilization individually.

Inference APIs that aggregate them do not have to be.

This dynamic is inherent to the API model itself, not specific to any single provider.

Why This Matters Strategically

This distinction explains several broader conclusions in the report:

• What segments of the market API-based inference versus reserved capacity captures long-term

• Why utilization variance dominates performance and tokens/sec variance

• Why inference APIs are structurally different from reserved-GPU platforms

It also clarifies why inference APIs optimized for aggregation and cost discipline are not simply “worse versions” of platforms optimized for model novelty or research adjacency. They are solving different problems and, critically, exhibit different utilization dynamics.

Enterprise infrastructure is entering a bifurcated cycle: an immediate scramble to operationalize AI under extreme resource constraints (Act 1), followed by a structural rewrite of organizational architecture itself (Act 2). The firms that win the next decade will bridge these worlds — capturing momentum now while accumulating long-duration advantage as AI becomes the primary consumer and producer of work.The evolution of enterprise infrastructure can be viewed as a two-act play.

Below are the key needs and constraints defining each act.

Act 1:

• Supply-side constraints (GPU scarcity, infra fragmentation).

• Tools that accelerate building (acceleration, security, orchestration).

• Many will be temporary arbitrage plays; some will become rails.

Act 2:

• A structural rewrite of “how work happens.”

• Systems treat agents as organizational primitives.

• Entire SoRs (AD, HRIS, ERP, IdPs, ticketing) get replaced.

Act 1 is the AI-native momentum phase: a period of acute infrastructure bottlenecks, new AI fabrics, and a proliferation of tools for not just building/securing but accelerating/unblocking AI-native applications and processes. Many of these bets may likely not endure into meaningful companies but could be good investment outcomes nonetheless.

Act 2 represents the structural transformation phase: where firms that dominate Act 2 will emerge not from “micro” rewrites using AI, or tooling where the foundational layers can eat into over time, but from foundational redefinitions of “how work happens” AND “lessons learned” from Act 1 that help create more enduring systems — imagine an enterprise org hierarchy not just being defined by employees but also agents as first class organizational elements - entire systems of record such as HR and core infrastructure services such as Active Directory will need to be reimagined.

So the strategic opportunity for us lies in balancing these acts — making targeted investments across both acts, balancing near-term momentum with long-term structural vision. This involves a 3 pronged approach:

1. Act 1 Investments: These investments will focus on the “tracks” of the current AI build out. Opportunities in this category are driven by immediate, intense demand and clear market dislocations, such as the ongoing GPU shortage. The key discipline here will be to differentiate between durable infrastructure plays and temporary arbitrage opportunities that will fade as the market matures.

2. Act 2 Investments: These investments will target “category creation” opportunities that fundamentally rewrite existing software and security paradigms. These are higher-risk, longer-horizon ventures that require a deep conviction in a company’s vision to see beyond the limitations of the current stack. (eg ..?)

3. The Bridge (Act 1+2): We also believe there will be valuable investments in companies whose Act 1 products are necessary to solve today’s pressing problems but are architected to become the system of record for the Act 2 paradigm. These are the platforms on which we will “double down,” as they offer the potential for both near-term revenue and long-term, defensible market leadership (eg Theom to not just replace current data classification, data lake protection solutions but also as a foundational layer for enterprises to rely for on a data governance platform to unlock agentic deployments).

   

II. Looking ahead to Act 2 structural transformation and working backwards into Act 1 and Bridge Opportunities

Some themes are proposed in the visual for discussion with some details on a few below:

Democratization of AI Clouds

Today’s compute economy is defined by scarcity — the “GPU-constrained” world.

Neo-clouds (CoreWeave, Nebius, Lambda Labs, Crusoe, etc.) and cloud fabrics that stitch across hyperscalers have emerged to address this shortage. These players capture short-term, ?-margin value in solving supply-side inefficiency while setting the stage for longer-term abstraction layers. We first wrote about this phenomenon in 2023 with “The Financialization of the Data Center” piece. Some will make it to Act 2, others won’t.

The next step will come from eventual democratization of GPU supply (e.g. multiple GPU makers, valid non-GPU options such as TPUs becoming mainstream etc), but also from virtualization and workload orchestration that lets developers treat heterogeneous compute as fungible while also serving local markets with a latency sensitive orientation. Sovereignty concerns will also play into the possible democratization of “country specific” GPU clouds.

We also believe future acts will be defined by heterogenous compute — across other types of GPUs, TPUs, ASICs and the CPU. This shift will have massive derivative effects across enterprise infrastructure.

LLMs ⇒ World Models

This is currently a deeply discussed and contested discussion in the AI community. Current LLMs possess a remarkable ability to process and generate text, but they lack a true, underlying understanding of cause and effect. The next frontier in AI research is the development of “World Models.” A world model is an internal simulation of an environment that allows an agent to predict the consequences of its actions before taking them and builds upon the mathematical underpinnings of Markov Decision Processes (MDPs) which is a model sequential decision-making in situations with uncertain outcomes. Formally, it learns the transition function of the environment, or the probability of ending in a new state s’ given the current state s and an action a, expressed as P(s’|s,a).

Recent research has shown that today’s LLMs can be “induced” to perform basic world model functions, such as determining the preconditions for an action and predicting the resulting state change. This is a foundational step toward creating agents that can truly reason, plan, and operate in complex, dynamic environments.

Some example areas of interest include:

• Advanced Simulation Environments: Platforms capable of generating vast quantities of high-fidelity synthetic data, which will be essential for training models to understand complex cause-and-effect relationships in a controlled manner. Simulation-first companies include Waabi, Applied Intuition etc and Synthetic Data Platform companies include Synthesis AI, Datagen, Sky Engine AI etc.

• Data Platforms for Multimodal Reasoning: The real world is not just text. World models will need to ingest and correlate information from a wide array of sources, including video streams, sensor data from IoT devices, and structured logs from enterprise systems. Investing in the data platforms that can unify and process this multimodal data is a tangible way to invest in the future of world models. Categories here include Physical Video and Sensor platforms

Robotics/IoT data platforms with companies like Viam, Formant etc. Even a next gen storage can be envisioned here for storage platforms purpose built for large scale simulations and multimodal training.

The real enterprise unlock is not synthetic environments — it’s stable, predictable agent behavior. World models are the missing reliability layer.

Agents ⇒ Things

Act 2 extends the agentic wave into the physical domain — robots, sensors, and “things” with agency. This transformation moves AI from desktops and servers into real-world decision loops — factories, logistics, and healthcare. Emerging players like Sanctuary AI, Figure, and Covariant represent the early prototypes of this convergence. These “things” may require a new hardware stack to enable them and the thinking will likely be split between on device and in the cloud.

• Edge inference + local autonomy ← critical for sovereignty + latency

• A hybrid stack: on-device reflexes + cloud-level planning

Act 2 physical systems will adopt a layered control stack: reflexive inference at the edge, and world-model-driven planning in the cloud.

Micro ⇒ Chunky SaaS

Workflows themselves will be rewritten from first principles. Instead of AI co-pilots within human-centric systems, Act 2 SaaS will be AI-first orchestration platforms that treat humans as optional participants in the loop. This could produce AI-native ERP, finance, and HR systems — startups like Doss and Quanta hint at this future.

Chunky SaaS collapses dozens of micro workflows into autonomous processes that produce outputs, not dashboards. If Act 1 gave us copilots, Act 2 gives us “self-driving back-office systems.”

III. The Bridge - areas where Act 1 monetization builds into Act 2 defensibility:

We discuss at depth the Agentic Cyber and Agentic Lifecycle themes in other pieces.

A brief overview of the AI Networks theme is included below (a detailed introduction into this topic is here.

• Why: Almost all clouds are at <50% GPU utilization with the primary bottleneck for scaling AI being the network. Traditional networking was built for statistical multiplexing (many small, independent jobs), but large-scale AI training operates as a single, massive, synchronized job across tens of thousands of accelerators, demanding all-to-all communication where the worst-case latency dictates job completion time.

• What: The solution is to evolve from a reactive, “best-effort” network to a proactive, deterministic, and “perfectly scheduled” fabric. The goal is to make the network as predictable as an on-chip interconnect, removing random delays and jitter that stall expensive accelerators.

• How: This is achieved through a co-design of hardware and software including next gen data center components such as Attotude, Enfabrica and discrete “sub systems” that offload transport and congestion control from the CPU, nanosecond-level time synchronization across the data center, allowing the network to precisely schedule every single data packet and ensuring a predictable, ultra-low-latency, and near-flawless fabric

V. Summary

We believe some of the most successful investments will capture Act 1 economics while accumulating Act 2 optionality.

The winners will exhibit a unique trait: Act 1 capital efficiency, Act 2 architectural inevitability.

Below you can see an example described by Sierra:

Why the ADLC Exists

The Agent Development Lifecycle (ADLC) defines how autonomous systems move from prototype to production, establishing the reliability and governance foundation for the agentic era. Just as the Software Development Lifecycle (SDLC) industrialized software engineering, the ADLC is doing the same for intelligent agents.

Below are a few key areas that are emerging as key parts of the ADLC stack:

The central insight: enterprises will pay for reliability, control, and visibility over autonomy—not for intelligence alone.

The last wave of agent frameworks proved agents could plan and act but failed to deliver operational reliability. They were stateless, brittle, and non-deterministic, unable to meet enterprise standards for safety, compliance, and interpretability. The next generation—anchored by Databricks Agent Bricks, OpenAI Agents SDK, and Temporal—brings new foundational infra for durability and control.

Winning companies will:

1. Abstract complexity into deterministic, explainable primitives.

2. Integrate with enterprise control planes for identity, policy, and observability.

3. Monetize reliability, not model intelligence.

The ADLC represents the industrialization of AI autonomy: a shift from building agents that can think to building infrastructure that ensures they can be trusted.

Side note: Greg seems to have adopted Jensen Huang’s signature Tom Ford leather jacket look for the announcement. The leather jackets are a signature look for Jensen as he reportedly owns over 50 of them and wears them almost exclusively at public events. The look has evolved into a brand symbol—so much so that analysts often call NVIDIA launches “the leather jacket show.” Jensen seems to appreciate strong American engineering and have to admit, it’s a sharp look!

Scale check

• 10 GW ≈ 4–5M GPUs (depending on racks and efficiency). That’s effectively Nvidia’s entire 2025 production and nearly double last year’s shipments.

• When Jensen Huang called the deal “monumental,” it wasn’t marketing—it was literally about the industrial scale of the project.

Financing geometry

• Nvidia invests $10B in equity for every 1 GW OpenAI deploys, up to $100B at 10 GW.

• For every ~$35B of GPU orders, Nvidia chips in ~$10B of its own stock. Net effect: OpenAI pays ~71% in cash, ~29% in equity.

• In plain terms, Nvidia is partially financing its own sales—a structure that ensures demand while tying OpenAI closer to its roadmap.

Breaking from Microsoft

This scale-up also highlights a deeper shift: OpenAI loosening its dependency on Microsoft.

• Microsoft pushed hard to block OpenAI’s Windsurf acquisition, worried it would threaten GitHub Copilot’s moat.

• The partnership has grown tense: Microsoft wants OpenAI locked into Azure, while OpenAI wants flexibility to chase cheaper, more abundant capacity.

• By cutting deals with Oracle for cloud and Nvidia for capital + silicon, OpenAI is deliberately hedging away from Microsoft’s grip.

The “enemy of my enemy” logic

Nvidia is equally motivated and pulling a Michael Corleone to seize power. The big three clouds—Amazon, Google, Microsoft—are all investing in their own silicon to escape Nvidia’s margins. So Nvidia has gone the opposite way:

• Prop up challenger clouds like Oracle, CoreWeave, Nebius, Crusoe.

• Seed alternative capacity pools that keep demand diversified and hyperscaler leverage in check.

• In that light, Nvidia and OpenAI’s alignment is classic: the enemy of my enemy is my friend.

The capital loop

The money flows almost like a cartoon: OpenAI spends with Oracle → Oracle buys from Nvidia → Nvidia invests into OpenAI → repeat. On paper it looks circular, even bubble-like—capital bouncing between the same three names.

But the distinction is that each spin leaves behind physical assets: GPUs shipped, racks powered up, substations wired, models deployed to hundreds of millions of users. It’s not just numbers recycling across balance sheets—it’s dollars hardening into infrastructure.

The historical rhyme

This isn’t tulips or banner ads. The better analogies are railroads in the 1800s or fiber buildouts in the early 2000s. Overcapacity wiped out equity holders in the short term, but the infrastructure remained and became foundational.

Risks

• Power access: interconnect queues and substations are the true bottleneck now, not chips.

• HBM supply: memory packaging could limit throughput regardless of how many GPUs are bought.

• Utilization: 10 GW of idling silicon would crater returns; keeping GPUs busy is existential.

• Policy: tariffs, permitting, or AI regulation could reshape the economics overnight.

Investor lens

• Watch LCOC (levelized cost of compute): who can turn megawatts into tokens cheapest?

• Vendor capital as moat: Nvidia financing its own demand gives it stickiness no startup chip vendor can replicate.

• Second-order bets: optics, HBM, advanced packaging, switchgear, immersion cooling—all benefit from this cycle no matter who “wins” the model race.

Bottom line

OpenAI’s 10 GW plan isn’t just capex—it’s a bid for independence from Microsoft, fueled by Nvidia’s need to weaken the hyperscaler triopoly. The capital loop may look like a meme, but it’s also a flywheel: capital → compute → models → demand → back to capital.

The question isn’t whether the cycle makes sense—it’s who will still be standing when the music slows.

dbt’s leadership is so strong that it has begun to commoditize the EL layer: if transformation is standardized in dbt, ingestion and loading can become utilities, not differentiators. That is a testament to the power of the dbt ecosystem.

dbt’s Lead: From Community to Compiler Intelligence

In January 2025, dbt Labs reinforced its technical moat by acquiring SDF Labs. The deal brought a Rust-based SQL compiler that enables dbt to truly understand SQL—not just execute it.

This acquisition cemented dbt’s role as the innovation engine in transformation, ensuring it remains central to the workflows of hundreds of thousands of analytics engineers.

Fivetran’s Move: Validating Transformation with Tobiko

Yesterday, September 3 2025, Fivetran announced the acquisition of Tobiko Data, the creators of SQLMesh and SQLGlot. SQLMesh and SQLGlot have already gained strong traction in the open-source community, and we highlighted them in our earlier Data Infrastructure in 2025 piece.

Fivetran, long known as the market leader in automated ingestion, now generates more than $300M ARR and is IPO-ready. By acquiring Tobiko, it signaled that owning transformation is essential to staying relevant in the modern stack—a direct validation of the standard dbt established.

What Tobiko adds for Fivetran:

• SQLMesh for incremental transformations and semantic awareness.

• Virtual Data Environments for safe, isolated development.

• SQLGlot for multi-dialect parsing across engines.

Tobiko’s founding team—Toby Mao (Airbnb/Netflix), Iaroslav Zeigerman (Netflix/Apple), and Tyson Mao—built SQLMesh based on years of experience with brittle pipelines at scale. Their vision of efficient, stateful transformations resonated with open-source users and Fivetran alike.

It’s important to note: this isn’t a zero-sum rivalry. dbt continues to lead transformation with unmatched community adoption shaping how data teams work. Fivetran’s acquisition of Tobiko is an acknowledgment—recognizing that transformation is the layer that truly matters, and that dbt defined the space.

Conclusion

Transformation is no longer the overlooked step between ingestion and dashboards. It is the center of the modern data stack, where governance, AI-readiness, and developer workflows converge.

• dbt Labs remains the anchor: the first name data teams reach for, the company pushing the boundaries with compiler intelligence and AI-assisted development.

• Fivetran is expanding its scope: Tobiko strengthens its offering, but more importantly, validates the centrality of dbt’s vision.

In the years ahead, dbt’s influence will only grow as AI and governance demand semantic understanding of data pipelines. Fivetran’s move underscores what the market already knew: transformation—pioneered and defined by dbt—is where the real value lies. Tobiko brings excellent technology to the Fivetran platform, but dbt continues to set the standard others follow.

Major congratulations to the Tobiko Data team on the validation—and credit to dbt Labs for creating the standard others are racing to follow

Source: Bloomberg and Census Bureau [link]

Intro: Data Centers Surpass Office Construction in 2025

This year, annual spending on new data centers is set to overtake general office construction. 10 years ago, US data construction was a paltry $2B compared to over $33B of office construction spend. After a decade of consistent growth up until COVID, office development has sharply declined from its 2020 peak, while data center investment has surged—driven by the AI arms race. This divergence underscores a profound shift in commercial real estate priorities: the physical backbone of the digital economy is becoming more valuable than the spaces where people work in person. AI’s insatiable appetite for compute, storage, and low-latency connectivity is reshaping the built environment at unprecedented speed.

1. Real Estate

The first key input to the build of the AI data center is real estate. The footprint of AI-driven data centers is scaling dramatically, from 275M sq ft in 2025 to 700M sq ft by 2030. The footprint of AI data centers looks more like a crypto mining data center than a traditional data center.

The real estate footprint of AI data centers is radically different than the traditional data center defined by Equinix and population density. Facilities are also getting physically larger, with hyperscale operators like Azure building campuses exceeding 2M sq ft—over 100× the size of a typical U.S. data center. AI data centers optimize for cheap land and cheap electricity, while traditional data centers optimized for bandwidth and proximity to population centers. CoreWeave’s 450K sq ft facilities illustrate the new “AI-native” build standard, optimized for high-density racks and liquid cooling. This expansion will demand favorable zoning, proximity to high-capacity fiber routes, and access to reliable power sources. The result: data center real estate is becoming one of the most competitive and strategic asset classes globally, representing more than 30% of Blackstone’s flagship fund. We wrote about this trend in 2023 and the financialization of the data center has only become stronger.

Notable facts:

• Blackstone now owns more data center square footage than some of the largest REITs, following its acquisitions of QTS and stakes in Digital Realty.

• DigitalBridge reports that AI tenant deals are often signed with prepayment or multi-year upfront commitments—rare in traditional colocation.

2. Capital

Data center development has become deeply financialized, with M&A volume hitting $73B in 2024. Private equity and infrastructure funds like Blackstone, KKR, and DigitalBridge are aggressively consolidating colocation providers, betting on multi-decade AI infrastructure demand. AI-native operators—CoreWeave, Crusoe, Lambda—are attracting both equity and debt financing to expand capacity quickly. Capital intensity is enormous: annual investment is projected to grow from $450B in 2025 to $1.2T by 2030. For investors, the blend of long-term leasing, mission-critical status, and AI-driven growth makes data centers a premier infrastructure play.

Notable facts:

• Private equity buyers are paying EV/EBITDA multiples in the mid-teens for hyperscale-ready assets, a premium over historical averages.

• AI-native operators like CoreWeave have secured multi-billion dollar credit facilities backed by GPU collateral, effectively making compute a financeable asset class.

3. Compute

The compute layer is entering an unprecedented growth phase, with chip shipments set to grow from 14M in 2025 to 70M by 2029. Only capital rich companies have the necessary compute to compete in AI, leading to a fragmentation of GPU rich and GPU poor companies. Aggregate AI compute capacity will surpass 200 exaflops by the end of the decade. Nvidia dominates the current revenue pool—GPUs represent the bulk of a projected $400B in AI chip sales by 2027—while TPUs, ASICs (i.e. by cloud providers), and AI-optimized XPUs (i.e. by Broadcom) also gain share. This growth is driving massive supply chain and manufacturing investments, from advanced packaging to high-bandwidth memory. In the AI era, data center competitiveness is increasingly a function of compute density per megawatt.

Notable facts:

• Nvidia’s current lead in AI accelerators gives it ~80% market share in data center GPUs, with gross margins exceeding 75%.

• HBM (High Bandwidth Memory) production is a key supply constraint—Samsung, SK Hynix, and Micron are all racing to expand capacity, with HBM prices up >30% YoY.

• By 2027, an AI training cluster capable of 1 exaflop may cost $500–$800M in hardware alone, not including networking or facilities.

• Arm-based CPUs are quietly gaining share in inference-heavy AI clusters due to lower power draw per token processed.

4. Energy

AI’s energy needs are accelerating at a scale that will reshape national grids. By 2030, AI workloads could consume 10% of total U.S. electricity, up from just 4% in 2023. Meeting this demand requires adding 90GW of capacity between 2025 and 2030—equivalent to building dozens of new nuclear plants or thousands of large-scale solar farms. AI power spending will nearly triple from $58B in 2025 to $191B by decade’s end. This dynamic makes energy sourcing and sustainability not just ESG talking points, but competitive differentiators in AI infrastructure.

• The AI power ramp—an extra 90GW by 2030—is roughly equal to adding the total current generation capacity of the United Kingdom.

• Data center PUE (Power Usage Effectiveness) averages ~1.2 for hyperscale, but AI training loads can push it higher due to extreme cooling needs.

• Some operators are exploring direct nuclear supply—Microsoft has hired nuclear engineers and filed for SMR (Small Modular Reactor) licenses.

• Curtailment agreements with renewable providers are being renegotiated to prioritize AI data center loads over less profitable grid exports.

5. Networking

High-performance networking is the circulatory system of AI data centers, with the total AI networking TAM projected to hit $100B by 2030. Leading players include Arista as the pure independent and NVIDIA, which has quietly grown its networking business to >$10B of sales after its acquisition of Mellanox. Optical transceivers alone will grow from a ~$12B market in 2024 to $40B by 2030, driven by the shift to 800Gb and above interconnects. The volume of 800Gb ports is expected to surge from 13M in 2025 to 284M by 2029, enabling both scale-out and scale-up AI clusters. Attotude’s low-latency interconnects, a $35B market by 2029, will be critical for tightly coupled GPU workloads. As models scale, the bottleneck shifts from compute to moving data fast enough between chips, nodes, and sites.

• The leap from 400Gb to 800Gb optical links can reduce model training time by double-digit percentages in tightly coupled GPU clusters.

• Co-packaged optics, still <5% of the market today, could surpass $15B TAM by 2030 as switch ASIC power budgets become limiting factors.

• The Infiniband vs. Ethernet battle is intensifying: Nvidia’s Quantum-2 Infiniband dominates training clusters, but cloud providers are pushing Ethernet-based RDMA to cut costs.

• Retimer and cable costs can account for >20% of the total networking bill in AI facilities—an often-overlooked capex driver. See our Astera Labs S1 review.

Enter the agent infrastructure stack.

If software was about deterministic instruction execution, and ML infra was about data pipelines and model tuning, agent infra is about managing stochastic, semi-autonomous systems that learn, act, and retry—without a human in the loop.

This is not an extension of existing infra. It’s a clean-slate redesign. Below are the five categories defining the stack.

🔧 1. Agent Builders: The New Interface Layer

Think LangChain, CrewAI, and Sema4. These frameworks abstract away the glue logic and let teams stitch models, tools, memory, and triggers into goal-driven agents.

Most enterprise prototypes today are built on glue-code sprawl: brittle chains of prompts, scripts, and Zapier-style logic. Agent builders aim to replace this with something composable, observable, and production-ready.

🧠 Strategic Insight: This layer will be hard to displace once embedded. Expect 2–3 dominant winners, each with distinct abstraction levels (think: Unix → Python → React).

⏱ 2. Persistence Engines: Durable Orchestration for Non-Determinism

Traditional workflow orchestrators (Airflow, Temporal) assume deterministic step execution. Agents violate that assumption. Their logic branches, retries, and re-plans on the fly.

That’s why companies like Zipline, Inngest and Hatchet are gaining traction. They act as a queue/cron + state store + retry engine—purpose-built for stochastic systems.

🧠 Strategic Insight: This category is inevitable. It will underpin any agent that runs longer than 15 seconds or involves a multi-step workflow. Bonus: strong data flywheel from agent execution traces.

🧠 3. Memory: Statefulness as Infrastructure

Current-gen agents are stateless. Each task begins with a fresh prompt. That’s broken. The move toward long-term memory will define whether agents feel brittle or adaptive.

Emerging vendors like Pinecone, Mem0 and Zep are trying to build structured memory layers—handling context, conflict resolution, summarization, and relevance—all with low latency.

🧠 Strategic Insight: Memory is both a UX unlock and an enterprise liability. Expect standards (e.g., memory protocols) to emerge for auditing, compression, and conflict resolution. This is a greenfield category but risks being swallowed by incumbents or embedded into agent frameworks.

🌐 4. Web & Tool Interaction: Killing RPA for Good

APIs are not enough. Most enterprise tools still rely on human GUIs. Agents need to control UIs programmatically. That’s the role of headless browsers + browser-native agent protocols like MCP.

Platforms like Browserbase let agents interact with SaaS interfaces as if they were humans. This is where RPA should have gone.

🧠 Strategic Insight: This is a linchpin for real-world integration, and it’s already splintering into protocol vs. execution layers. Whoever wins here will become the Zapier + Puppeteer for agents.

🔐 5. Agent Identity & Auth: No Enterprise Rollout Without This

Today’s agent security posture is terrifying. Tokens are hardcoded. Agents “borrow” user IDs. There’s no audit trail or permission boundary.

Agent-native auth systems like Composio and Arcade are building scoped, verifiable identities for agents—think service accounts + RBAC + session tracing + audit logs.

🧠 Strategic Insight: If you’re selling to a CISO, this is the entry point. Long-term, it becomes the enforcement layer for trust and compliance across the stack.

🧠 The Control Plane War Has Begun

Inference is now a solved problem—API pricing is dropping, quality is improving, and access is ubiquitous. The bottleneck has moved upstream. What’s missing is an execution substrate that can coordinate stochastic, goal-seeking agents at scale.

That substrate is the agent control plane: the orchestration, memory, auth, and tooling layers that allow autonomous agents to act reliably across real-world systems. Today, no one owns it. There’s no standard runtime, no dominant memory protocol, no consensus on agent identity or benchmarking.

This is the power vacuum. Whoever wins the control plane will define how agents are built, deployed, and governed—just as Kubernetes defined modern app orchestration. The biggest AI infra companies of this cycle won’t compete on inference. They’ll own the scaffolding that makes inference useful.