What replaces the single chip is a routed inference stack: each request flows through a FLOPs-rich accelerator for prefill, hands the KV cache to a memory-bandwidth-rich accelerator for decode, and streams the output back. The diagram below is the architecture; the rest of this piece is why the physics demand it and what the market consequences are.

The Two Phases

LLM generation is autoregressive: every output token depends on every token that came before it. That dependency creates two computational regimes.

Prefill processes the input prompt. Every prompt token is known up front, so the model computes their key/value vectors in parallel — one large matrix multiply that builds the KV cache. Compute-heavy. FLOPs-bound. High arithmetic intensity, on the order of 100–400 FLOPs per byte read.

Decode generates output tokens one at a time. Each step streams the full model weights and the entire KV cache from memory to multiply against a single activation vector, then appends one new K/V pair to the cache. Bandwidth-heavy. Low arithmetic intensity, roughly 1–2 FLOPs per byte.

Same model. Same hardware. Two different workloads.

The Metrics That Decide Deployment

Two latencies decide whether an inference deployment is usable.

• Time to first token (TTFT) is set by prefill. Interactive UX breaks above roughly 3 seconds.

• Time per output token (TPOT) is set by decode. Interactive targets sit at 100–300 ms per token, at minimum matching reading speed.

API pricing reflects the asymmetry. Output tokens cost meaningfully more than input tokens at every major provider — GPT-4o lists $2.50 per million input tokens against $10.00 per million output tokens, a 4x premium on the memory-bound phase. Decode is where the margin lives.

Why One Chip Can’t Do Both Well

Modern GPUs are designed around HBM-fed tensor cores. Prefill uses both. Decode uses neither well — the tensor cores idle while HBM saturates streaming weights for a single output token at a time. Plot it on a roofline and the two phases sit on opposite sides of the knee.

Inference engines have spent the past two years working around this. vLLM’s chunked-prefill scheduler interleaves prefill chunks with decode steps so compute and memory bandwidth saturate simultaneously on the same GPU. TNG Technology Consulting reports the technique increased total token throughput by approximately 50% in their production vLLM deployment serving Llama-3.1-8B (source). The lesson is structural: even on one chip, the operator gets paid to treat prefill and decode as different jobs.

The silicon-level unbundling is the same trick at scale.

Nvidia + Groq: FLOPs Buys Bandwidth

Nvidia owns prefill. Hopper and Blackwell are HBM-rich and FLOPs-rich, and they run the dominant share of frontier-model context processing. They do not own decode economics. HBM is supply-constrained — three vendors (SK Hynix, Samsung, Micron) — and decode burns bandwidth that could otherwise serve training.

The relevant Nvidia HBM specs:

Each generation buys more bandwidth, but the underlying architecture is still HBM-fed tensor cores. Decode still loads the full weight matrix from external memory for every token.

Groq is the architectural opposite. No HBM. The LPU runs entirely from on-die SRAM, which is faster per byte than HBM by at least one order of magnitude and is not bottlenecked by the HBM3e supply chain. The chip does not win on raw FLOPs. It wins on bandwidth-per-dollar, where bandwidth is the constraint — i.e., decode.

Acquiring Groq lets Nvidia route prefill to Blackwell and decode to LPUs over high-bandwidth fabric. The HBM bus stops being the bottleneck on output-token economics. Nvidia’s HBM exposure on inference workloads tightens. The strategic prize is freeing HBM supply for training, where it has no substitute.

Cerebras + AWS: SRAM Meets Trainium

Cerebras is the wafer-scale extreme. The WSE-3 is a single 46,225 mm² die — roughly 57x the silicon area of an H100 — with 44 GB of on-chip SRAM. A 70B-class model and its KV cache fit on-die. Decode runs at SRAM speed. Cerebras has built its commercial pitch around sustained tokens-per-second on memory-bound inference, and the architecture maps directly onto the decode workload.

What Cerebras lacks is fleet-scale prefill capacity. CS-3 systems are scarce, capital-intensive per unit, and optimized for memory-bound work. Long-context prefill — increasingly the dominant workload as agentic and reasoning models drive context windows toward 1M tokens — needs cheap, abundant FLOPs.

AWS Trainium fills the gap. Trainium2 is a high-FLOPs, HBM-equipped accelerator built for compute-bound work. The integrated Cerebras–AWS stack runs prefill on Trainium fleets, hands KV state to Cerebras CS-3 systems, and serves decode at wafer-scale bandwidth. AWS monetizes Trainium silicon on a workload where Nvidia GPUs already dominate. Cerebras gets the prefill volume the WSE was never designed to deliver.

Mirror-image strategy to Nvidia/Groq, same underlying thesis.

The Economics

Three forces make the unbundling structural rather than a clever optimization.

1. Workload mix is shifting toward decode. Reasoning and agentic models output substantially more tokens per request than chat-style models did 18 months ago. A growing share of frontier-model inference revenue comes from decode rather than prefill.

2. HBM is the binding constraint on decode capacity. HBM3e production runs near full utilization, with 2026 supply already pre-allocated. Any decode workload that migrates to SRAM-based silicon frees HBM for training and prefill, where the alternatives are worse.

3. Decode carries the margin. At a 4x price premium over input tokens, decode is the gross-margin lane. Operators serving decode on lower-cost silicon capture the spread directly. A disaggregated stack lowers cost per output token versus a single-architecture deployment — the exact figure depends on workload mix, but the sign is unambiguous.

What This Changes

1. The HBM oligopoly’s pricing power weakens at the margin. Decode migration off HBM-dependent silicon slows HBM3e demand growth and reshapes HBM4 timing assumptions.

2. Inference orchestration becomes a control point. The router that decides which phase runs on which silicon — and tunes batch size, chunk size, and KV-cache transfer policy — is the new margin layer. The technical primitives already exist inside vLLM (chunked prefill, prefix caching, KV-cache offloading). The step from “interleave on one GPU” to “route across heterogeneous accelerators” is small. Nvidia (Dynamo, post-Groq), AWS (Bedrock + Trainium + Cerebras), and independent inference platforms (Together, Fireworks, Baseten) will fight for it.

3. Capital intensity per token declines. A unified GPU stack overpays for FLOPs on decode and overpays for HBM on prefill. The disaggregated stack pays for each only where it is consumed.

4. New silicon entrants get a clearer wedge. The single-architecture incumbency advantage erodes when the workload itself is two distinct workloads. Memory-bound silicon (Groq, Cerebras, d-Matrix, Tenstorrent) gets a defensible decode lane. Compute-bound silicon keeps prefill but loses its decode rent.

Bottom Line

Prefill and decode are different physics. Different roofline regimes. Different cost structures. Different margin profiles. Inside a single GPU, scheduler-level disaggregation already pays. Across silicon, Nvidia/Groq and Cerebras/AWS are the same trade — pair a compute engine with a memory engine, route the workload, capture the spread.

The single-accelerator era is over. The routing layer owns the margin.

Security has the same structure. Did the exploit land? Did the agent reach the flag? Did the defender detect and contain it before exfiltration? Binary, automatically verifiable, and infinitely repeatable in a sandbox. The missing ingredient has been the environment infrastructure to produce these signals at scale. That is now being built. Anthropic is already publishing research with companies like Incalmo on breaching systems with AI.

The Drone Parallel

Cost asymmetry has become the defining strategic problem of kinetic warfare:

A $20K munition forcing a $4M interceptor breaks defense economics. The industrial response — Anduril, Epirus, directed energy programs at the primes — is fundamentally a race to push defense cost-per-engagement toward $1–$10 (lasers, microwave, EW).

Cyber is the same problem with worse asymmetry:

• AI attacker compute per engagement: $10–$100

• Average enterprise security spend: $5M–$50M+ annually

• Average breach cost (IBM 2024): $4.88M

• Open-source attacker tooling: $0

How do you stop a $10 autonomous exploit without spending $1M per incident? That is the question the next decade of security spend will be organized around.

What Is Different From Drones

Three things make the cyber version harder:

1. Software has no kinetic bottleneck. Drone attackers need factories. Cyber attackers need a checkpoint and GPUs.

2. The defense surface is unbounded. Airspace is finite. Modern enterprise attack surface — identities, SaaS, supply chain, model weights, agent tools — is not.

3. Attribution is weak. Deterrence-by-punishment underwrites kinetic doctrine. It does not apply in cyber.

The implication: the AI defender must be cheaper, faster, and more autonomous than its kinetic equivalent. A human SOC analyst at $200K fully loaded handling 10–20 investigations per day cannot match an attacker running thousands of episodes per hour.

The Training Bottleneck

You cannot RL-train an agent in production. The analog to a unit test or a proof checker is a sandboxed environment with realistic topology, verifiable objectives, adversarial structure, and parallel scale — thousands of episodes per hour. This is the picks-and-shovels layer for security RL. Whoever supplies the environment captures economics from every downstream model.

Implications

• Offensive AI commoditizes faster. Assume open-source parity with commercial tooling within 24 months.

• Defensive AI consolidates around whoever owns the best environment. That is where the training data lives.

• Legacy SIEM/SOAR is structurally mispriced. A 6 figure/year log platform cannot defend against a $10/hour autonomous attacker. Expect a generational vendor reset within 36 months.

• Capital flows in order: environments first, defender models second, deployed agents third — mirroring how drone-defense capital rotated from platforms to payloads to fire control.

Summary

Verifiable-reward RL fell math and code in under 24 months. Security has the same reward structure and a larger economic surface. The drone analogy gives the cost curve; the lesson is that whoever industrializes the reward environment first determines who wins the next decade of the attacker-defender game.

AI inference workloads require 10,000–1,000,000x the compute of a comparable SaaS transaction, depending on model size and task complexity. This shifts value from software toward compute supply chain owners, compressing gross margins at the application layer. Hyperscalers are spending a combined $700B in 2025–2026 capex — roughly 5x the 2022 baseline — because AI-native workflows represent a step-function increase in compute demand over the SaaS baseline they replace.

What determines the actual production of compute-hours? Most discussion focuses on GPUs. But compute-hours are the output of a system with 7–10 interdependent inputs. Each layer is tightly coupled. When constraints stack, supply elasticity collapses.

Key Trends

1. Nvidia sells full racks, not GPUs. The unit of compute has shifted from chip to rack. GPU count per rack moves from 72 (Hopper) to 576 (Rubin) — 8x in one generation. Rack-level memory is scaling faster: GB200 NVL72 at 30 TB HBM3e, GB300 NVL72 at 40 TB, Vera Rubin NVL144 projected at 100 TB — roughly 3x within the Blackwell generation alone. Per-GPU memory follows the same trajectory: H100 at 80 GB HBM3, B200 at 192 GB HBM3e (2.4x in a single generation), with HBM4 at Rubin pushing further. Every layer of the stack — power, cooling, interconnect, cabling — gets redrawn when the reference architecture moves.

2. The interconnect question is unresolved. Scale-up (within rack) and scale-out (rack-to-rack, pod-to-pod) have diverging physical layer requirements. Copper and AECs remain competitive at short reach on cost, reliability, and power. Co-packaged optics improves bandwidth density at distance but carries a power premium over copper today. Where the crossover lands determines which interconnect vendors are structural winners.

3. Memory is a primary constraint. HBM capacity growth is outpaced by model size and context length expansion. Supply is concentrated across three vendors. HBM accounts for roughly 35–40% of AI server bill-of-materials cost, making memory economics a system-level concern, not a component-level one. Any component that reduces memory pressure — near-memory compute, KV cache compression, sparsity — commands disproportionate ROI in the current supply environment. Per-GPU memory progression: H100 (80 GB) → H200 (141 GB, 1.76x) → B200 (192 GB, 1.4x) → Rubin (288 GB est., 1.5x) — roughly 3.6x over six years.

4. ASICs are in production, not pilot. Google TPUs and AWS Trainium are at scale. Each requires its own compiler toolchain, interconnect topology, and systems integration. The infrastructure stack can no longer be optimized solely around NVLink/NVSwitch.

5. Energy is an architectural input, not an operational variable. Rack power density has increased from 10–20 kW (CPU era) to 100–130 kW (GB200), with 200+ kW expected at Rubin densities. Optical transceivers consume roughly 2–3x more power per port than copper at equivalent bandwidth — a material tax at 200+ kW/rack. Decisions that appear to be performance tradeoffs are simultaneously energy budget decisions.

Scale-Up Clusters: Why NVIDIA’s Architecture Wins on Durability and Efficiency

GPU density per rack has increased roughly 6x from 2021 to 2026 — from 8–16 GPUs per rack to 72+ in the GB200 NVL72, with the NVL144 and NVL256 roadmap targeting continued doubling. This is a systematic architectural shift, not incremental improvement, and it compounds into durable cost and performance advantages.

The Communication Hierarchy: Why Staying In-Rack Is Structurally Cheaper

Cluster communication cost and latency increase sharply as data crosses physical boundaries. The hierarchy, from lowest to highest cost: (1) in-package and NVLink — sub-microsecond latency at 900 GB/s per GPU in the NVL72; (2) rack-scale interconnect — copper-viable but bandwidth-constrained; (3) cross-rack networking — where latency and cost increase nonlinearly. Every inter-rack hop adds microseconds of latency, consumes optical transceiver power (~2–3x per port versus copper at equivalent bandwidth), and introduces fabric congestion risk. Larger in-rack clusters reduce total communication overhead and deliver lower cost-per-token at scale. The math is structural.

NVIDIA’s Structural Advantage: NVLink as a Moat

NVIDIA is the only vendor currently able to deliver a validated, high-density scale-up cluster at rack scale. The NVL72 ships with 900 GB/s GPU-to-GPU bandwidth via NVLink 4, 30 TB of pooled HBM3e memory, and a single-rack power envelope of ~120 kW within NVIDIA's reference architecture. Competitors must traverse PCIe or Ethernet for equivalent multi-GPU communication — 10–20x lower bandwidth per link than NVLink at equivalent rack density. AMD's Infinity Fabric does not scale beyond 8 GPUs without external switching. Google TPUs and Amazon Trainium are ASIC-scale designs without a comparable proprietary high-bandwidth fabric. NVIDIA's inference cost-per-token advantage at 72-GPU density is structural, not a function of current market position.

Efficiency and Durability Implications

First, eliminating inter-rack hops removes optical transceiver power overhead — material when each 800G optical port draws ~10 W versus ~4 W for equivalent copper.

Second, unified HBM memory pools across 72 GPUs allow workloads to avoid KV cache eviction and model sharding penalties that dominate inference latency at smaller cluster sizes.

Third, higher GPU utilization: the GB200 NVL72 sustains higher model flops utilization on large-batch inference workloads than disaggregated clusters of equivalent total GPU count, because all-reduce operations stay within NVLink rather than crossing fabric.

NVLink all-reduce is 4–8x faster than InfiniBand at equivalent cluster size. As inference workloads scale, the performance gap between NVLink-based and non-NVLink architectures widens. The NVL72 roadmap to NVL144 and NVL256 compounds CUDA ecosystem lock-in with physical architecture lock-in.

Investment note: The industry-wide push toward larger scale-up clusters requires exponentially more cables, higher signal integrity requirements, and greater pressure on copper and optical interconnect at short reach as GPU density per rack increases. This trend is net positive for high-bandwidth interconnect specialists addressing signal integrity challenges at 72+ GPU densities — including Attotude.

Where we are today: massive shortages

The underappreciated risk is that a single bottleneck shuts down the entire system. Current materials under extended constraint: copper, aluminum, indium phosphide; power and electrical equipment; laser components; HBM memory; CoWoS packaging capacity at TSMC. Most are booked out 2+ years. Removing one constraint shifts pressure to the next layer — which is why aggregate GPU supply numbers mask the actual delivery reality.

Structural Constraints — Severity, Bottleneck, and Timeline

Core Thesis

The AI infrastructure stack is not just GPU-constrained — it is system-constrained. Removing one bottleneck shifts pressure to the next. Companies that own chokepoints in this stack — advanced packaging, power delivery, optical interconnect, laser components — hold structural leverage analogous to what ASML holds in semiconductor manufacturing.

Four structural forces are reshaping the stack:

1. Slow-scaling inputs versus fast-scaling demand. Power generation and high-voltage transmission scale on decade timelines. Compute demand scales on 12–18 month cycles. The gap between these curves is the primary investment surface.

2. Layer-specific bottlenecks force architectural transitions. When a layer hits a physical limit, the supplier map is redrawn. Copper-to-optical in networking is the live example — CPO share expected to rise from near zero to ~36% of addressable ports.

3. Silicon is fragmenting. AMD, Google TPUs, Amazon Trainium, and OpenAI custom silicon are eroding NVIDIA’s training monopoly. Value migrates toward scarcer upstream inputs when the compute layer commoditizes. NVIDIA remains the only vendor selling the full rack.

4. Compute is geography-bound. Power availability, transmission access, and regulatory environment are making AI infrastructure decisions location-specific. Texas has more planned data center capacity than any other state. Site selection is now a primary capital allocation decision, not a secondary one.

Recent M&A

The M&A velocity in optical and interconnect is the strongest forward indicator of where supply constraints are most acute.

• Groq —> Nvidia $20B, inference chip

• Celestial AI → Marvell ~$3.25B

• Optical connectivity | Enfabrica → NVIDIA ~$900M

• Ayar Labs ($1B+ Series D) — likely next optical exit

$4.15B in optical-adjacent exits in under 12 months. NVIDIA’s $2B strategic investment in Lumentum and Coherent (March 2026) is a direct acknowledgment that laser supply — specifically electro-absorption modulated lasers inside every 800G and 1.6T transceiver — is a compute constraint. Lumentum controls 50–60% of global EML supply.

Public Market Exposure — Where to Express Today’s Needs

Five Key Structural Observations

1. Optical fabric is the highest M&A velocity category.

$4.15B in exits in <12 months. Ayar Labs ($1B+ Series D) is the next likely event. The laser layer is the least visible and most acute constraint within optical: electro-absorption modulated lasers (EMLs), the component inside every 800G and 1.6T transceiver that converts electrical signals to light, are undershipping demand by ~30%, with all capacity locked under long-term agreements and incremental units priced at a premium. Lumentum controls 50–60% of global EML supply; Nvidia’s $2B strategic investment in Lumentum and Coherent (March 2026) is a direct acknowledgment that laser supply is a compute constraint.

2. CoWoS is the rate-limiting step for everything upstream.

GPU supply, custom ASICs, and AI accelerators all flow through TSMC CoWoS at ~20K wafers/month. Independent packaging alternatives and panel-level interposers are the highest-leverage supply chain investment in this cycle.

3. Power is a software opportunity.

Grid interconnection, energy OS (1,700+ utilities, multiple types of power <> data center), and workload-aware power management are software wedges into a hardware-constrained market. Vertiv captures the hardware layer; the software orchestration layer is open.

4. Inference economics are separating from training economics.

Training is NVIDIA’s domain. Inference is not. Cost/token, memory efficiency, and latency matter, and they are not optimized by training-oriented GPU architectures. Critically, LLM inference, image generation (fal.ai, Flux), and video generation (Runway, Sora) have distinct compute profiles — model-type-specific accelerators may be a more tractable bet than general inference silicon.

5. NVIDIA is vertically integrating; build around it.

Enfabrica (fabric), Mellanox (networking), Cumulus (switching). Companies must build on technology NVIDIA does not control: Ethernet AI fabric, RISC-V, photonics, advanced packaging. Partner, don’t compete.

The severity of the supply-demand mismatch is visible not just in procurement data but in production reliability. Anthropic — among the most compute-intensive AI companies in the world — operates below 98% uptime, a figure that would be disqualifying for any enterprise software vendor. AWS S3 targets 99.99%; Salesforce and Workday routinely exceed 99.9%. Anthropic’s reliability ranks among the lowest of any major software service, not due to engineering failure but due to compute scarcity. It’s reported that much training resources in 2026 will be diverted to keep inference online, ultimately slowing model releases.

Despite concerns that further generations of Nvidia hardware would cause heavy depreciation on older models like H100s, prices have hardly come down and even become more scarce over time:

Frontier AI labs are diversifying their compute base. The most consequential shift is not a single provider’s growth — it is the pattern of AI labs actively decoupling from single-cloud dependency. OpenAI is diversifying away from exclusive Microsoft reliance toward OCI, which absorbed a material portion of its recent backlog growth. Anthropic is expanding beyond AWS and GCP into FluidStack, Crusoe, and Oracle. These are strategic decisions by the two largest AI labs to distribute infrastructure risk across providers. OCI’s backlog growth is real, but its concentration in OpenAI creates bilateral risk: OpenAI has extended itself aggressively across multiple infrastructure commitments and may restructure obligations if its own revenue trajectory compresses.

Even Microsoft cannot keep up. Azure’s $625B AI-era backlog represents +1,150% growth over its pre-AI baseline — and it is still not enough. Microsoft has spent the past year signing capacity deals with CoreWeave, Crusoe, Nebius, and NScale, sourcing a rapidly growing percentage of incremental compute from third parties. This is not a strategic pivot — it is a supply constraint forcing structural dependency.

The neoclouds are real, though still small. CoreWeave, Nebius, NScale, FluidStack, and Crusoe entered the table from zero, combining for $131B in new commitments — enterprise-scale, multi-year GPU reservations from customers and hyperscalers alike who cannot build fast enough to serve demand internally. In practice these remain small relative to the $100B+ investments made by hyperscalers. Today neoclouds represent 10–20% of capex in AI data centers; hyperscalers account for 80–90%.

The Capital Ceiling

The four largest hyperscalers — Microsoft, AWS, Google, and Meta — are collectively committing approximately $700B in AI infrastructure capex in 2026, a figure that approaches or exceeds 100% of their operating free cash flow and creates negative overall free cash flow. Boards impose discipline; buybacks, dividends, and core business investment compete for the same capital.

The hyperscalers will not reduce capex programs, but they are unlikely to materially exceed the FCF ceiling on a sustained basis. The marginal dollar of new capacity will not come from Microsoft or AWS — it will come from neoclouds financed by private credit, infrastructure funds, and sovereign capital. They entered the commitments table from zero not because hyperscalers are retreating, but because hyperscalers cannot grow fast enough to serve demand at the margin.

In less than three years, Microsoft transitions from a fully self-supplied infrastructure operator to one sourcing roughly half its incremental compute from third parties. For Microsoft, CoreWeave is not a partner — it is a crutch. For CoreWeave, Microsoft is not a customer — it is validation that the largest infrastructure spender in the world cannot meet its own demand.

Why This Is Structural, Not Cyclical

Compute-hours are produced by a chain of interdependent inputs. The binding constraints have migrated from silicon to physical infrastructure: power availability, high-voltage transmission, transformer manufacturing (lead times: 2–4 years for large units), cooling, and grid interconnection. Demand scales on 12-to-18-month cycles; electrical infrastructure scales on decade-long ones.

That mismatch is structural scarcity. When one constraint is relieved, pressure shifts to the next layer serially — GPU supply exposes the power bottleneck; power exposes transformer lead times. The result is geography-bound compute: AI infrastructure is forced into specific regions by power availability, land, water access, and regulatory environment. OCI and CoreWeave have been structurally more aggressive on power procurement than AWS and Azure at the margin, and the backlog data reflects it.

Networking follows the same pattern: copper interconnects are hitting limits at 10,000-plus GPU cluster scale, forcing a transition to co-packaged optics (CPO) that collapses a supplier tier and elevates a new set of component manufacturers into structurally critical positions.

Implications

Every hyperscaler is growing — and none are growing fast enough. These are extraordinary growth rates by any historical standard. Yet neocloud backlog emerged from zero in the same period. The demand signal is not that hyperscalers are failing; it is that AI compute demand is scaling faster than even trillion-dollar balance sheets can absorb. GCP’s +382% growth suggests Google is executing better on AI-native positioning than its installed base would imply.

OCI is the most important cloud story to watch — and possibly the most exposed to ROIC failure before any neocloud is. Oracle recently eliminated approximately 30,000 employees, inconsistent with scaling into a large build commitment. A reported Texas site failure raised delivery timeline questions. Most consequentially, Larry Ellison has reportedly pledged Oracle equity as collateral against personal debt tied to a significant media acquisition, creating forced-selling exposure and board independence risk at exactly the moment the company requires sustained capital discipline.

Oracle (ORCL) — LTM Cash from Operations: $22.3B / CAPEX: $35.5B / Free Cash Flow: -$13.2B. Source: Bloomberg

The conventional risk framing treats neoclouds as the fragile capital structures and hyperscalers as the durable ones. In OCI’s case, that framing may be inverted. A purpose-built neocloud with no founder pledge overhang, no legacy workforce to restructure, and no unrelated media liability may be a more reliable 5-year counterparty. OCI’s backlog growth is real; whether Oracle has the organizational and financial architecture to deliver it is an open question.

Neoclouds are the marginal capacity provider. CoreWeave’s $41B, Nebius’s $20B, and Crusoe’s $20B are the overflow valve for a market growing faster than any single balance sheet can absorb. The neocloud capital structure — private credit, infrastructure funds, sovereign capital — is purpose-built for exactly this role.

Regionalization is the next structural theme. AI infrastructure is geography-bound by power availability, data sovereignty, and national industrial policy. NScale ($14B valuation) is the primary neocloud partner for Microsoft and OpenAI in Europe. Neysa is executing the same playbook in India. The Middle East and Latin America are next — sovereign AI programs, abundant energy, and governments subsidizing local infrastructure to avoid permanent compute dependency on US providers. Expect regional neocloud champions in each geography, underwritten by hyperscaler offtake agreements and sovereign capital.

Risks

Risk 1 — Proprietary Silicon at Scale

The one credible path by which hyperscalers recapture internal workloads from neoclouds is success with custom ASICs. AWS (Trainium/Inferentia) and Google (TPUs) are the only hyperscalers with viable proprietary silicon in production today — but at relatively small scale relative to their total compute footprint. Microsoft, Oracle, and Meta have no viable ASIC. OpenAI is working with Broadcom on custom silicon but is at minimum three years from meaningful production. Meta’s response is instructive: rather than build internally, it has committed to AMD at scale — reportedly one of the largest GPU procurement agreements in the MI300X generation — wagering that a third-party accelerator can substitute for internal silicon. Until AWS and Google demonstrate ASIC scale that materially reduces neocloud dependency, the ASIC threat remains a 5-year risk, not a present one.

Risk 2 — Demand Deceleration

The one scenario in which neoclouds face existential pressure before hyperscalers do is a rapid cooling of frontier AI demand — a world in which model capability improvements plateau, enterprise adoption stalls, or capital markets withdraw from AI infrastructure financing. Neoclouds carry the most concentrated exposure: GPU lease obligations are fixed, cost structures are capital-intensive, and there is no diversified revenue base to absorb a demand shortfall. Hyperscalers, by contrast, can redirect capacity to general cloud workloads and absorb losses within broader P&Ls. A demand collapse would reprice neocloud equity and debt simultaneously, and several newer entrants — underwritten on aggressive utilization assumptions — would face covenant stress before a single hyperscaler earnings call reflected the problem.

The evidence does not support near-term cooling. Anthropic and OpenAI are not experiencing outages because they are poorly run — they are experiencing outages because demand is outrunning every unit of capacity they can procure. These are rationing events in a supply-constrained market. GPU depreciation curves have proven slower than prior hardware cycles; H100 utilization rates have remained elevated and secondary market pricing has held above what prior GPU generations sustained at equivalent ages. Until the outage pattern reverses, a demand deceleration severe enough to threaten neocloud economics remains a tail risk, not a base case.

Project Rainier, Anthropic and AWS

Anthropic has built what is today the most diversified and cost-efficient compute architecture among frontier AI labs. OpenAI remains almost entirely dependent on Nvidia. Microsoft’s internal chip program is years behind schedule. The structural implications favor Anthropic on unit economics and negotiating leverage as inference workloads scale. While Anthropic has had so much demand, it has struggled with up-time — its long-term strategy is the most fundamentally resilient.

One important caveat up front: compute advantage amplifies model advantage; it does not replace it. If a competitor’s models are materially better, customers absorb the higher token cost. The argument here is not that Anthropic wins because of infrastructure. The argument is that equivalent model quality delivered at 30–60% lower cost per token is a compounding advantage — on margin, on training budget, and on the pace of iteration.

The Structural Divide

Every frontier model company faces the same binding constraint: various estimates put Nvidia at ~90%+ of the discrete GPU market and a dominant share of AI training accelerators deployed in the field, and that pricing power is real. H100 on-demand list pricing at major hyperscalers (AWS, Azure, GCP) peaked at $7–$8/hour; even after the 2025 oversupply correction drove spot and commodity-broker rates to $1.50–$3.00/hour, the published on-demand rate at hyperscalers still runs $6–$9/hour. Committed and reserved pricing falls somewhere in between — typically $2–$5/hour depending on contract length and volume — but requires capital-intensive multi-year lockup.

The question is not whether you can afford Nvidia GPUs. At scale, the question is whether Nvidia is the only lever you can pull, and what that dependency costs you annually.

There is a layer beneath the GPU pricing question that gets less attention: HBM supply. The real bottleneck in high-density AI clusters is not the accelerator die — it is High Bandwidth Memory, packaging, and power density. TSMC’s CoWoS packaging capacity is a genuine constraint, and Nvidia’s pricing power is partly a downstream effect of controlling where that supply gets allocated. Any chip independence strategy that bypasses this layer — custom accelerators still require HBM stacks — is incomplete. True chip independence means secured HBM allocation, packaging slots, and power-deliverable data center capacity; the die is just the most visible part. Anthropic’s TPU and Trainium commitments are meaningful partly because Google and Amazon have negotiated their own HBM supply chains, transferring that allocation risk away from Anthropic.

Anthropic answered the dependency question structurally. OpenAI has not.

AI LAB COMPUTE STRATEGY (March 2026)
──────────────────────────────────────────────────────────────────────────
Lab           Primary Stack          Diversification        Custom Silicon
──────────────────────────────────────────────────────────────────────────
Anthropic     TPU v5/v6 + Trainium2  Multi-hyperscaler      Operational now
              + Nvidia (research)    operational today

OpenAI        Nvidia H100/H200/B200  None at scale today    Broadcom: 2027+

Microsoft     Nvidia (dominant)      Maia 200: 2 sites      Maia 200: 2yr late
──────────────────────────────────────────────────────────────────────────

The strategic conclusion this table points to: the winning economic model is not chip ownership — it is deep integration into a hyperscaler’s silicon program. Labs that pursued partnership-driven vertical integration have operational compute today. Labs that pursued ownership are years behind.

Anthropic: Dual-Hyperscaler, Dual-Accelerator

Project Rainier

In October 2025, Amazon opened Project Rainier — a purpose-built AI compute campus in Indiana with an $11 billion infrastructure investment. The cluster runs on Trainium2, AWS’s custom AI accelerator, and operates at an estimated scale that makes it among the largest operational AI compute clusters in the world. Anthropic is its primary tenant; training Claude is the primary workload.

The economics are the point. Trainium2 runs at approximately half the price of comparable Nvidia H100 instances in sustained workloads. Effective committed cost drops to roughly $0.50/chip/hour — compared to $2–$5/hour for H100 on committed or reserved contracts at equivalent hyperscaler rates. Anthropic has reported 50% cost reductions and material throughput gains versus GPU configurations on specific training runs.

Amazon’s total investment in Anthropic now stands at $8 billion (following the initial $4 billion in 2023 and a further tranche in late 2024). This is not a vendor relationship. It is a strategic alignment that gives Anthropic priority allocation on the Trainium roadmap and infrastructure access at cost.

The Google TPU Deal

On October 23, 2025, Anthropic announced an expansion of its Google Cloud partnership committing to 1 million TPUv7 Ironwood chips — over 1 gigawatt of compute capacity — coming online in 2026. The total deal value is approximately $52 billion. The structure is split: 400,000 TPUv7 chips purchased directly from Broadcom for roughly $10 billion in finished racks (Broadcom CEO Hock Tan confirmed the order publicly), with the remaining 600,000 chips rented through Google Cloud Platform at an estimated $42 billion in remaining performance obligations. Anthropic deploying purchased TPUs in its own facilities — rather than renting all capacity through GCP — signals a shift toward compute ownership, not just procurement. Google has invested $3 billion in Anthropic at approximately 10% equity, with an additional $1 billion committed in January 2025.

To house the direct-purchase TPU clusters, Anthropic selected Fluidstack — a UK-based data center operator — to build custom facilities in Texas and New York, part of a broader $50 billion domestic infrastructure commitment announced in late 2025. Fluidstack handles site construction, cabling, burn-in, and acceptance testing; their CEO described the partnership as enabling “rapid delivery of gigawatts of power” for frontier AI development. The sites are coming online throughout 2026, creating approximately 800 permanent jobs and 2,400 construction jobs.

TPUv7 Ironwood offers approximately 20–30% better raw performance than prior generations, and 50% better price-per-token at the system level versus equivalent GPU configurations. Google cites 60–65% lower power consumption per equivalent FP8 throughput versus GPU alternatives, which matters at gigawatt scale.

The realistic cost advantage for TPUs on LLM training workloads is 30–60% versus H100 clusters in comparable configurations. Claims of 4–10x cost improvement are achievable but apply to specific workloads with highly optimized kernels — not general production workloads. TPUs require custom compiler stacks (XLA), and most model teams optimize for CUDA first. The migration complexity is real: TPU software tooling, debugging infrastructure, and workload portability have historically lagged the Nvidia ecosystem by years. Anthropic’s engineering investment to operate across multiple architectures is a genuine cost that partially offsets the hardware savings, and the thesis depends on Anthropic having solved — or continuing to solve — those tooling gaps at scale.

Why Multi-Accelerator Is Structurally Distinct

Anthropic runs across three accelerator families: Google TPUs, AWS Trainium2, and Nvidia GPUs for research and CUDA-dependent workloads. Each platform is assigned to configurations where its price-performance profile is strongest.

Most frontier model companies rely on a single hyperscaler and a single accelerator family. Anthropic’s portfolio approach produces three advantages that are difficult to replicate quickly.

Cost arbitrage is the most obvious: the blended rate across TPUs and Trainium2 versus an all-Nvidia stack is 30–60% cheaper at comparable throughput on optimized workloads. Supply redundancy is the second: a capacity constraint at one vendor does not halt training. Negotiating leverage is the third, and arguably the most durable: when you are simultaneously committing to 1 million TPUs and a gigawatt-scale Trainium cluster, you have a credible alternative in every procurement conversation. Nvidia’s pricing power is real — but only against buyers with no alternative. It is worth noting that the mere existence of Google TPUs as a credible alternative was reportedly sufficient to save OpenAI — which has never deployed TPUs at scale — roughly 30% on its own Nvidia chip negotiations. Anthropic, which actually uses them at production scale, has considerably more leverage.

The total committed compute across Google TPUs and AWS Trainium now exceeds 2 gigawatts of dedicated capacity. On publicly available estimates, Anthropic’s annualized revenue run rate was in the range of $3–5 billion by late 2025, trending higher. The compute cost per token on this architecture is structurally lower than an Nvidia-only stack at similar scale.

OpenAI: Nvidia Dependency and an ASIC That Hasn’t Shipped

OpenAI’s production compute fleet runs on Nvidia GPUs. Published estimates place active GPU deployments at 100,000–200,000 H100/H200 chips, with additional Blackwell (GB200) orders in progress. These figures are not publicly confirmed. What is confirmed: Bloomberg estimated OpenAI’s quarterly Nvidia GPU spend at approximately $1.8 billion (roughly 6% of Nvidia’s total quarterly revenue at the time). At that run rate, Nvidia dependency is not a footnote — it is the primary cost structure.

Training compute spend in 2024 was approximately $3 billion; inference spend was approximately $1.8 billion. GPT-4 inference alone was projected at roughly $2.3 billion in 2024 — approximately 15x the original training cost. This asymmetry is the core inference economics problem: as models scale and usage grows, inference spend dominates, and the cost structure is entirely Nvidia’s.

Sam Altman has stated OpenAI would own “well over 1 million GPUs” by end of 2025, anchored by the $500 billion Stargate infrastructure program. But compute diversity within Stargate is less developed than the headline suggests.

The Broadcom ASIC: Production Starts H2 2026

On October 13, 2025, OpenAI and Broadcom announced co-development of custom AI accelerators targeting 10 gigawatts of capacity. OpenAI designs the architecture; Broadcom manufactures through TSMC. Production begins H2 2026. Meaningful deployment at scale is a 2027 event at the earliest.

For competitive purposes, this chip does not exist yet. OpenAI’s inference economics through at least 2026 are Nvidia economics. OpenAI also signed a $10 billion deal with Cerebras in January 2026 for 750 megawatts of low-latency inference compute through 2028 — more immediately actionable, with Q1 2026 first deployments, but narrow in scope.

The Stalled Nvidia Mega-Deal

In September 2024, Nvidia and OpenAI announced a letter of intent for a partnership in which Nvidia would invest up to $100 billion in OpenAI as OpenAI deployed at least 10 gigawatts of Nvidia systems.

As of March 2026, no definitive agreement has been signed. Nvidia CFO Colette Kress stated in December 2025 — two months after the announcement — that the $100 billion investment was “never a commitment.” The Wall Street Journal reported negotiations went on ice after internal Nvidia skepticism about OpenAI’s business model. No money has changed hands. Jensen also framed the investment as an invitation to invest from OpenAI, showing OpenAI is essentially at the mercy of Nvidia:

The circular economics were always awkward: Nvidia investing in OpenAI while OpenAI is its largest customer. The stall suggests the announcement was more strategic signaling than procurement commitment.

Microsoft: Announced, Not Shipped

Microsoft’s Maia chip program is the clearest example of the gap between custom silicon ambition and deployment reality.

Maia 100, announced November 2023, was deployed almost exclusively for internal Microsoft staff productivity workloads. It was not designed for generative AI inference at production scale. Two years post-announcement, it has not powered any significant external production service. Microsoft spent $31 billion on Nvidia AI chips in 2024 alone — the most of any cloud provider — while Maia 100 sat on the bench.

Maia 200, announced January 26, 2026, is a more serious design: TSMC 3nm, competitive FP8 throughput, 216GB HBM3e. On paper it is capable. In practice it entered limited deployment in two data centers in early 2026, over two years behind the original roadmap. Delays were caused by last-minute design changes, simulation instability, and high employee turnover affecting chip teams. By the time Maia 200 reaches meaningful Azure deployment, Nvidia’s Blackwell generation will be well-established in production across the industry.

The most telling data point is not the delay itself — chip design is genuinely hard, and two-year slips are not unusual. It is that Microsoft committed $31 billion in a single year to the vendor it was trying to reduce dependence on, while the alternative produced nothing deployable. Custom silicon timelines are measured in years, not quarters. Microsoft is not going to be chip-independent within this model generation cycle.

The Economics at Scale

At committed rates across a gigawatt-scale cluster, the blended cost advantage of TPU + Trainium over all-Nvidia runs roughly $1–2 billion per month on comparable throughput. Annualized, that is $12–$24 billion in structural cost advantage at full 2-gigawatt deployment — before accounting for power efficiency differentials, which are material at that scale. This assumes high utilization and workload fit; realized savings depend on kernel maturity and scheduling efficiency, both of which require sustained engineering investment to sustain.

Inference is projected to account for two-thirds of all compute spend in 2026, up from one-third in 2023. Every token served is a cost event. The cost per token is the unit economics of every API-facing AI business. Anthropic’s blended architecture makes that unit economics structurally better than a lab running equivalent inference on H100s — not by an order of magnitude, but by enough to matter at scale.

Implications

For Anthropic: The compute architecture is not just a cost advantage — it is a capacity advantage. At 2+ gigawatts of committed compute, Anthropic can run training runs that competitors cannot finance at equivalent cost. That compounds model iteration speed. The critical risk is engineering complexity: maintaining efficient workflows across three accelerator families and two compiler stacks requires real investment and creates coordination overhead that Nvidia-only stacks do not.

For OpenAI: The Broadcom ASIC is a credible long-term move, but it does not affect 2026 economics. As H100 spot prices have collapsed, OpenAI’s inference costs have improved — not because of anything OpenAI did, but because oversupply hit the GPU commodity market. That dynamic will tighten again as Blackwell deployment absorbs capacity through 2026. The one scenario that meaningfully narrows the gap: if Blackwell (GB200) delivers on its performance claims, inference cost per token on Nvidia hardware could fall 40–60% from H100 baselines, which compresses Anthropic’s current advantage. That is a real variable to watch.

For Microsoft: Maia 200 arriving two years late is a failure of execution, not strategy. The strategy was correct. But the more important signal for investors is the gap between ambition and output: $31 billion to Nvidia in one year while the internal program produced nothing deployable.

For investors: Compute cost structure is not typically a line item in AI company analysis. It should be. The cost per token served is the unit economics of every API-facing AI business. A 30–60% structural advantage at the compute layer does not translate 1:1 into margin — energy, networking, and engineering overhead all factor in — but it translates substantially. Anthropic’s path to positive gross margin is structurally easier from this compute foundation than from an all-Nvidia stack. At the scale these companies are targeting, the difference is not rounding error.

For the silicon market: The Anthropic model — dual-hyperscaler, multi-accelerator, committed long-term capacity — is likely to become the template for any frontier lab that reaches sufficient scale to negotiate it. Google and Amazon have strong incentives to subsidize Anthropic’s compute to validate their custom silicon against Nvidia at production scale; Anthropic gets below-market compute and Google and AWS get a credible reference proving TPU and Trainium can train state-of-the-art models. Ownership of chips is hard. Integration into the right silicon program is tractable, scalable, and — as this case shows — faster to gigawatt scale than any internal program has managed.

Summary

Anthropic has built the most cost-efficient and diversified compute architecture among frontier AI labs, anchored by Project Rainier — among the largest operational AI compute clusters in the world — and a commitment to 1 million TPUv7 Ironwood chips at 1+ gigawatt scale via a $52 billion deal structured across direct Broadcom purchase and GCP rental. Fluidstack is building the US data centers to house that compute. The blended economics are 30–60% cheaper per token than equivalent Nvidia H100 configurations on optimized workloads.

OpenAI remains structurally dependent on Nvidia through at least 2026. Its Broadcom ASIC is real but does not reach scale until 2027. The stalled $100 billion Nvidia investment deal is PR, not procurement strategy.

Microsoft’s Maia program is two years behind schedule. The company spent $31 billion on Nvidia chips in 2024 while its internal silicon produced nothing deployable for production.

Compute advantage amplifies model advantage. It does not replace it. But in an industry where inference spend is scaling faster than training spend, and where the cost of serving tokens will compound at every major model release, the lab that built the most efficient compute architecture 18 months ago is running a structurally different race than the one that didn’t.

We analyzed public commentary from leading AI operators and hyperscalers and mapped the bottlenecks they consistently cite across the stack.

These constraints arise from a tightly coupled stack:

Each layer has independent lead times and independent bottlenecks. When constraints stack, supply elasticity collapses. Even if GPU output doubles, effective compute-hours can remain scarce.

And that scarcity is not evenly distributed.

The Fiber Analogy Is Incomplete

There are clear parallels to the late-1990s fiber buildout:

• ~$500B in telecom infrastructure investment

• 39 million miles of fiber laid

• <5% utilization at the 2001 peak

• A decade to absorb excess capacity

But there are two critical differences:

1. Fiber could sit idle cheaply.

2. The AI buildout is funded by hyperscaler balance sheets, not junk debt.

Source: Coatue chart of the day

Fiber could sit idle at near-zero marginal cost.

AI infrastructure burns cash every day it sits unused.

• Power contracts

• Cooling costs

• Staffing

• Depreciating GPUs

Compute depreciates in ~5-8 years. Fiber depreciated over decades.

Which brings us to the most important constraint in the stack:

Power.

Power is the binding constraint in AI

Unlike GPUs:

• Power is location-bound.

• Interconnection queues take years.

• Transmission upgrades take longer.

• Generation requires long-term offtake.

• Permitting is political and regional.

You cannot spin up gigawatts the way you spin up GPU clusters. A GPU order cycle is measured in quarters. A transmission upgrade is measured in years.

In many U.S. regions, power availability is already the binding constraint on new AI campuses. Each major data center operator is rushing to lock up as much cheap power as they can.

Source: Public estimates

Last year we projected AI data centers would reach:

• 10% of U.S. grid demand

• ~145 GW by 2028

New bank estimates suggest we hit those numbers before 2027 .

That is not a cyclical constraint.

That is structural.

As an example, CoreWeave’s revenue backlog is scaling faster than its contracted power footprint. Commitments can scale in months. Power cannot.

Power Is Not a Commodity — It’s a Heterogeneous Stack

Deliverable megawatts — firm, high-density, grid-connected — are the scarce asset.

Electricity differs by:

• Baseload vs peaking

• Dispatchability

• Intermittency

• Transmission congestion

• Regulatory structure

• Geographic deliverability

A firm nuclear-backed megawatt co-located with transmission is not equivalent to intermittent solar behind congestion.

As AI campuses move toward multi-gigawatt scale, these differences become economically decisive.

The U.S. grid is not one market. It is ~1,700 fragmented micro-markets with political overlays.

For hyperscalers, that means:

• Tariff fragmentation

• Interconnection queue risk

• Congestion pricing exposure

• Regional political constraints

Further, there are geographical nuances. Texas has more new GW of data centers than all other states combined. Equity markets are re-rating toward regions with power headroom. Texas-heavy utilities (NRG Energy, Constellation Energy) and landholders (Texas Pacific Land) have become indirect AI beneficiaries. Access to power capacity, where it is, and what type of power is a key determinant of which energy players will thrive.

Source: Coatue chart of the day

The Inference Shift Changes Everything

Training is episodic CAPEX. Inference is ongoing load.

Inference-heavy facilities are provisioned with headroom for:

• Peak traffic events

• Failover redundancy

• Geographic routing

• Grid instability

• Future expansion

But utilization fluctuates:

• Model efficiency gains

• Traffic routing shifts

• Customer concentration

• Product cycles

The result: unused firm capacity.

This is the power analog to idle reserved GPU instances.

We’ve already seen compute marketplaces emerge to manage overbuild risk.

Power is next.

Power Is Becoming Financialized

Long-duration power contracts are starting to resemble financial instruments.

We are beginning to see:

• Subleasing of contracted power blocks

• Transfer of interconnection rights

• Monetization of demand flexibility

• Bundled “power + compute” offerings

The next logical step is standardized contracts, clearinghouses, and secondary liquidity.

This is where structural opportunity emerges.

The Strategic Example: Vertical Integration

Alphabet recently paid ~$5B to acquire Intersect Power (founded 2016), a large-scale clean energy developer.

Intersect develops and operates:

• Utility-scale solar

• Battery storage

• Hybrid generation

The acquisition vertically integrates generation with AI infrastructure — reducing basis risk, securing long-term supply, and internalizing power strategy.

This is not an energy investment. It is a compute hedge.

Power is no longer a utility expense. It is a competitive input.

The Venture Opportunity: The “Prime Broker” of Power

The market structure around AI power procurement is immature.

The bottleneck is measurable.

The durable venture layer is not steel and concrete. It is:

• Transaction rails

• Portfolio optimization software

• Structured finance

• Liquidity layers

What does not exist yet:

• A robust secondary clearinghouse for:

  • Capacity blocks

  • Interconnection rights

  • Long-duration contracted megawatts

If inference overbuild materializes, secondary liquidity becomes mandatory.

Why Power Scarcity Persists Even in Overbuild Scenarios

Here’s the key insight:

Even if GPUs overshoot demand by 5–10 years…

Power does not.

Power is:

• Regulated

• Capital intensive

• Politically constrained

• Slow to expand

Memory is cyclical.

Compute pricing compresses.

Power availability remains sticky.

Scarcity shifts pricing power upstream — into the physical layer.

The Structural Feedback Loop

AI lowers the cost of producing software and services.

Lower costs → more adoption.

More adoption → more inference.

More inference → more infrastructure.

More infrastructure → more power demand.

And the more this loop accelerates, the more valuable deliverable megawatts become.

What Happens If We Overbuild?

The central risk is not that AI demand disappears.

It is that parts of the stack overshoot by years.

But here’s the asymmetry:

• Idle GPUs can be redeployed.

• Idle fiber sits cheaply.

• Idle power commitments are expensive.

Compute overbuild compresses margins.

Power overbuild creates optionality.

Those are different regimes.

The Clean Thesis

Power is the lowest-variance, longest-duration constraint in the AI compute stack.

It is:

• Slow moving

• Structurally regulated

• Fragmented

• Financializing

• Becoming programmable

And unlike models or agents, it does not get commoditized by algorithmic improvement.

For venture investors obsessed with model layers, the more durable wedge may sit beneath them:

The transaction, orchestration, and financial infrastructure of energy.

Because wherever portfolios form, software follows.

AI infrastructure is becoming a power portfolio management problem.

And wherever portfolios exist — financial infrastructure follows.

In the early, pre-1990s days of the semiconductor industry, chip designers faced a structural dilemma.They had the technology to specify logic and build powerful custom integrated circuits, but without scalable simulation and verification, their work was more art than infrastructure. Progress was fragile and expensive, and the full system only “proved itself” once it was physically manufactured—meaning bugs were only discovered after fabrication. Eventually, electronic design automation (EDA) changed this by shifting correctness upstream, translating design into verified, executable systems through software.

A similar dynamic currently exists with today’s AI models. As AI shifts from chat to agents running real workflows, the limiting factor isn’t how smart the model is. The limiting factor, like with early chip design, is in whether we can reliably verify the work produced by AI models. Models can already write, browse, reason, and plan at a level sufficient for many professional tasks. But given the absence of reliable reward signals for the models—clear, consistent definitions of success across long-horizon workflows involving tools, judgment, policy, and taste—durable automation remains out of reach.

Enter reinforcement learning. Today, RL environments are playing the same role for AI agents that EDA played for silicon.They translate human intent into executable behavior by making success measurable at scale. But unlike EDA, they must also address how the “spec” for human labor is not deterministic. “Correct” is a moving target with many dimensions, and as agents improve, it becomes harder to measure success with a clean, dependable signal. As a result, RL environments must be living systems that evolve to resist reward hacking and encode judgment. This makes verification—not models—the true bottleneck to automation.

To be clear, EDA and RL aren’t exact analogs. EDA verifies outputs and can sometimes prove correctness; RL verifies incentives and, at best, bounds failure probabilistically. Still, the comparison is useful.A more recent analogy is perhaps the rise of Scale AI in the mid-2010s. At the time, most enterprises lacked in-house machine learning (ML) expertise, and cutting-edge models were still academically constrained. Early buyers wanted data for narrow, static tasks—bounding boxes, transcription, basic classification—primarily for “hard tech” domains, like autonomous driving. Scale’s core insight was that once models became capable enough, progress would be bottlenecked not by algorithms, but by data—and that a sophisticated buyer class would eventually emerge to demand it.

That inflection arrived with the transformer era (2018–2020). As foundation models like GPT-3 demonstrated surprising general capabilities, the constraint shifted again: from data volume to data complexity. The most advanced buyers—AI labs like OpenAI, DeepMind, and Anthropic—suddenly needed feedback over long horizons, as models were now tasked with reasoning, tool use, policy compliance, and multi-step workflows. One-off annotation no longer sufficed.

Over the past three years, this shift has accelerated. Models now need less raw data and far more experience—the kind that encodes enterprise workflows typically locked inside experts’ heads and the rise of expert data. Once data became feedback, labeling became verifiable work.

RL environments are the natural consequence of this transition. They transform workflows into simulated worlds where actions are observable, outcomes can be graded, and learning compounds over time. Environments and evaluations become the new datasets, and training becomes continuous rather than episodic. Replication training is to RL what internet-scale text was to LLMs: the mechanism by which scale produces generalizable infrastructure.

This reframes the competitive landscape. The critical question is no longer, “Who has the best model?” Rather, it’s “Who controls the infrastructure that determines what work is learnable at all?”

Verification Is the Hard Problem—and Solving It Requires Research Depth

Verification is where most RL efforts fail. Defining what “correct” means when there are multiple good answers, partial progress, or policy constraints is fundamentally a research problem, not a tooling problem.

Modern RL training pipelines, therefore, are no longer linear. Models critique humans, humans audit models, and algorithms arbitrate disagreement. The training signal emerges from the system, not any single actor.

As models improve, verification gets harder, not easier. Benchmarks saturate. Reward hacking appears. Graders drift. Environments must be refreshed continuously to stay training-relevant. This demands deep expertise in reward modeling, distribution matching, telemetry, and failure analysis.

As a result, the strongest RL-environment teams look less like SaaS companies and more like applied research organizations. They pair staff-level ML researchers, systems-oriented engineers, and forward-deployed builders with strong domain intuition. Code is rarely the constraint; judgment, data quality, and research velocity are.

Why Coding and Computer Use Are the First RL Markets to Scale

The earliest large-scale adoption of RL environments is happening in coding and computer use. These domains combine three properties that make them ideal early markets:

1. High economic value

2. Dense, machine-observable interaction traces

3. Relatively clean verification loops

Coding already has compilers, tests, and deterministic failure modes. Computer-use tasks—browsers, spreadsheets, CRMs, internal tools—produce replayable state transitions that can be graded and compared. This makes them well-suited for replication training of environments, where agents repeatedly execute long sequences until performance improves.

As a result, coding and computer-use environments are becoming the proving ground for agentic RL. Not because they are the endgame, but because they are the first domains where environment realism, reward design, and eval-to-production correlation can be demonstrated convincingly.

In practice, the deepest environments today are coding-based, such as terminal- and DevOps-centric environments. Coding is programmable because state, tools, and outcomes are explicit and replayable.

These markets matter because they establish trust: with research teams, with buyers, and with budgets. They are the wedge—not the destination.

Who is buying RL environments today?

The early buyers of RL environments look strikingly similar to the first buyers of autonomous-vehicle (AV) data. Just as Scale AI’s early revenue was overwhelmingly concentrated among a handful of AV leaders—Apple, Uber, Lyft, Tesla, and Google/Waymo—frontier AI labs are the dominant initial customers for RL environment companies. This is not a coincidence. RL environments sit upstream of capability gains, and the labs are the only organizations today with both the urgency and the budgets to pay for training infrastructure before it is fully productized. As with AV data in 2018, buyer concentration is high, contracts are large, and vendor selection is tightly coupled to research impact rather than procurement formalities.

Among labs, Anthropic appears to be the single largest buyer across both coding and computer-use environments. Across vendors focused on coding and computer use, Anthropic is estimated to be spending *on the order of* tens of millions annually on RL environments, making it the clearest anchor customer in the market today. Anthropic signed these contracts in 2025 and are already scaling rapidly; based on current trajectories, their aggregate lab spend on RL environments is likely to grow 3–5× into 2026 as environments move from experimentation to a core part of model training. OpenAI’s total spend is harder to isolate due to its broader data footprint, but it has already signed multiple seven-figure RL-environment and human-data contracts, indicating similar strategic intent.

Labs are also buying adjacent inputs aggressively, particularly human expertise for coding and long-horizon reasoning.

These purchases reinforce a critical point: labs are not just buying models or raw data—they are buying training throughput, and RL environments are becoming a first-class line item in that budget.

Enterprises, by contrast, are entering through a different door. Rather than buying standalone environments, they are purchasing forward-deployed, outcome-oriented solutions that embed RL environments under the hood. This is the story that Applied Compute is telling and why investors are so excited about its trajectory.

Product-led companies like Glean, Notion, and Cursor are running pilots or building in-house environments while evaluating vendors. This pattern mirrors earlier infrastructure cycles: labs fund the tooling when it is raw and research-heavy; enterprises adopt once it is wrapped around concrete workflows and measurable ROI.

This is why it is critical to have lab success in the early days to achieve escape velocity.

The implication is clear. RL environment companies will be built on lab money first, just as Scale was built on AV customers before expanding into enterprise ML and later LLM/AI research. Understanding which vendors are winning lab mindshare today is therefore the clearest leading indicator of who will control the training layer tomorrow.

The Real Prize: Long-Form White-Collar Work Across Many Environments

The long-term opportunity for RL environments lies in long-form workflows that span multiple tools, not single-step tasks. These workflows are how value gets created across finance, operations, go-to-market, and engineering.

A finance workflow might move from spreadsheet analysis to browser-based research, into an ERP system, and back into documents for approval and audit. An insurance claim touches intake portals, policy systems, historical databases, email, and compliance tooling. Legal work spans research databases, drafting tools, revision cycles, and risk review. In each case, success is judged by downstream outcomes—risk reduction, correctness under audit, or decision quality—not by any single action.

The same pattern holds across core operating roles. Sales moves across CRMs, email, calendars, pricing tools, and pipeline forecasting. Marketing spans analytics dashboards, ad platforms, creative tools, and experimentation systems. Customer support blends ticketing systems, product logs, documentation, and multi-turn communication. Developer workflows shift between IDEs, browsers, repositories, test runners, and CI systems. These are composed systems of environments, not isolated tasks.

In other words, these are not single environments—they are composed systems of environments. Training agents for them requires:

• Persistent state across tools

• Cross-environment credit assignment

• Verification tolerant of multiple valid paths

• Rewards tied to business outcomes, not toy metrics

This is where most early systems will break. One-off environments saturate quickly, and narrow benchmarks stop teaching. The future belongs to teams that can build environment factories: systems that assemble, orchestrate, and refresh multi-environment workflows continuously.

In this regime, progress comes from repetition at scale. Replication training—running the same workflow thousands of times across slightly varied environments—is how learning compounds. This is an emerging field in RL environments and will determine the market leader over time. The bottleneck is no longer model capacity. It is environment fidelity, verification quality, and orchestration scale.

The dynamics are shifting from building models to earning trust, access, and validation inside the frontier labs.

What does the winning RL Environment company look like?

Between 2026 and 2030, the RL environment market will narrow decisively. What today looks like roughly 20 seed- to Series A-stage companies on relatively similar footing—be they forward-deployed teams, early environment builders, or research-heavy startups—will resolve into three to five market leaders, with one to two dominant platforms pulling meaningfully ahead.

This is not a land-grab driven by environment count or early demos. It is a selection process driven by two reinforcing advantages:

1. Who earns “thought partner” trust from the frontier AI labs early, and

2. Who builds the research organization capable of industrializing replication training and verification.

Labs today are capturing low-hanging fruit in many simple environments to teach AI to use tools, especially to train v0 of computer use. In the future, labs will optimize less for the volume of environments shipped, as basic applications are trained. They will optimize for complexity, quality, and what comes next, and they will increasingly concentrate their spend on the few teams that help them push the frontier forward.

Here are six key principles of what will determine who breaks out in 2026-2030:

1. Frontier labs will choose embedded thought partners, not on-demand vendors

2. RL environments will evolve from brittle artifacts to automated infrastructure

3. Replication/environment training and verification define the research moat

4. Lab trust and research depth are self-reinforcing

5. Frontier lab work is the foundation; enterprise work is the multiplier

6. Depth in core and complex domains (i.e., coding) beats environment breadth (lots of environment apps)

1. Frontier Labs Will Choose Embedded Thought Partners, Not On-Demand Vendors

RL environments are currently still immature. Many are single-application, brittle to UI or workflow changes, and partly manual in how tasks are defined or graded. This is acceptable today, but it will not be as agent autonomy increases.

Agents can already run autonomously for two to three hours so long as the setting is constrained. As autonomy increases, training shifts from isolated tasks to long-horizon workflows spanning multiple environments, where state persists, decisions compound, and success only comes much later. This shift fundamentally changes what labs need.

As a result, frontier labs aren’t looking for vendors who can “build environments on request.” They’re looking for thought partners: teams who can help define how RL infrastructure must evolve as agents become more autonomous. In practice, this means co-designing new kinds of environments, stress-testing verification in ambiguous settings, and experimenting with replication training regimes that don’t yet have established playbooks.

Buyer concentration is a feature, not a bug. Winning a few anchor lab relationships matters far more than broad distribution. As a reminder, Scale AI had over 80% revenue concentration in roughly five AV customers going into 2019-2020.

‍Key Takeaway: Labs gravitate toward teams they trust, like working with, and treat as extensions of their own research organizations. Frontier labs are increasingly the gravitational center for RL talent. Startups that maintain live research pipelines into academia and labs inherit that gravity; others slowly fall out of relevance.

‍2. Environments Become Environment Factories: From Brittle Tasks to Automated Infrastructure

The technical evolution of RL environments is clear. RL environments will move from single-app, hand-curated tasks; annual or semi-manual grading; and fragile assumptions about tools and interfaces, toward multi-environment workflows; automated task generation and variation, and hybrid verification systems that scale with agent capability.

The end state isn’t “better environments.” It’s automated RL infrastructure: environment factories that continuously assemble, orchestrate, test, and refresh environments as agents improve and software ecosystems evolve.

This shift will be gradual. RL environments will stay largely manual for some time; if the entire pipeline were already automatable, frontier labs would simply build it in-house. The fact that they haven’t is precisely why this market exists.

Key Takeaway: As agents improve, environment quality—not environment count—becomes the binding constraint. Labs will reward teams that can absorb complexity, keep environments stable, and anticipate what the next generation of training will require.

3. Replication training and hybrid verification are the compounding technical moat

The core research challenge over the next five years is not simply building more RL environments, but turning environment training into a scalable system. This is where replication training becomes the defining primitive.

Replication training replaces bespoke, one-off environment building with controlled repetition at scale. Instead of constantly inventing new tasks, teams pick a small set of high-value, long-form workflows and run them thousands of times across slightly varied environments—different inputs, starting states, tool access, and constraints—while keeping the underlying workflow stable. Learning compounds not from novelty, but from repeated exposure across changing conditions.

Building reliable, extensible replication training is non-trivial. Replication training requires:

• Deterministic reset and replay, so workflows can be run repeatedly without drift

• Environment abstraction layers, which allows for variation without breaking realism

• Parallel orchestration, to run large numbers of long-horizon trajectories

• Instrumentation and telemetry, to surface where and why agents fail over time

The objective is to extract maximal learning signal from a minimal set of environments—driving down marginal cost while preserving fidelity.

Verification is the harder half of the system. In long-form, multi-environment workflows, there’s rarely a single correct answer, and as agents improve, the reward signals often become noisier, not cleaner. That’s why winning systems use closed, hybrid verification loops: experts score model outputs, models review expert judgments, humans supervise and correct models, and algorithms reconcile disagreement, detect drift, and curate edge cases

In this regime, data work becomes intellectual—the job is to construct and maintain a reliable training signal in the presence of ambiguity. Replication training only works if this verification loop remains stable as scale increases. That stability is a research problem, not a tooling problem.

Prompt optimization systems like GEPA are not alternatives to RL infrastructure, but adjacent primitives that plug into the same replication and verification systems.

The core skills required to build this system are rare and cross-disciplinary:

1. Systems engineering, to manage orchestration, replay, and scale

2. Applied ML research, to design rewards and diagnose failure modes

3. Human-in-the-loop design, to balance automation and judgment

4. Product intuition, to decide which variations actually teach the model something new

Teams that master these skills turn replication training into a compounding advantage—and define the real moat in RL environments.

Key Takeaway: Replication training only becomes a moat when it’s paired with a stable, closed-loop hybrid verification system—because the winner isn’t the team with the most environments, it’s the team that can reliably extract learning signal from a small set of long-form workflows at scale. That combo (repeatable training + compounding, ambiguity-resistant verification) drives down marginal cost, prevents drift, and turns every run into durable advantage.

4. Research Lineage Compounds: Trust → Talent → Frontier Exposure → Better Systems

These dynamics create a powerful flywheel, and research credibility is the entry ticket. Teams with genuine research lineage—often signaled simply by who built them—earn trust faster with both labs and investors.

Key Takeaway: Teams with strong research organizations win deeper lab engagements. Those engagements expose them to frontier failure modes earlier than the rest of the market. That exposure feeds directly into better replication training systems, stronger verification pipelines, and lower marginal cost per new environment. Over time, this compounds into durable technical advantage.

5. Lab-first builds the core; Enterprise scales the distribution

While lab trust, research depth, and talent reinforce one another, they are not equal inputs. If one must serve as the foundation for breakout success, it is unequivocally frontier lab work. This is where RL infrastructure is actually built. Enterprise work, by contrast, is where that infrastructure is applied, customized, and monetized.

Frontier labs force teams to solve the hardest problems first. Environments must generalize across models. Replication training must work at scale. Verification must hold up under long-horizon ambiguity. Orchestration, replay, and evaluation-to-performance correlation cannot rely on customer-specific shortcuts. Lab buyers are purchasing infrastructure primitives—environments, tasks, training loops—not bespoke outcomes. That pressure produces reusable systems rather than one-off solutions.

Enterprise demand today looks different. Most enterprises are not buying RL environments or training infrastructure directly. They are buying agentic applications tied to concrete workflows and KPIs, often delivered through forward-deployed engineering. This work is valuable, but inherently more custom. Enterprise agents benefit disproportionately from infrastructure that has already been hardened upstream in lab settings—effectively “RL menus” rather than bespoke training from scratch.

The asymmetry matters. Infrastructure built for labs lowers the marginal cost, risk, and time-to-value of enterprise deployments. Replication training pipelines, environment factories, and hybrid verification systems developed for lab use increasingly power enterprise work behind the scenes. Over time, what looks like bespoke enterprise delivery becomes configuration on top of lab-derived infrastructure.

Key Takeaway: Frontier lab work is the foundation. It produces the training and verification infrastructure that later makes enterprise deployments cheaper, faster, and more scalable. The teams that break out will be lab-first platforms that use enterprise demand as a multiplier—not the other way around.

6. Depth in Core, Complex Domains Beats Environment Breadth

In the early stages of building enduring RL infrastructure, depth in a small number of complex domains matters far more than breadth across many shallow ones. Not all environments are equally valuable. The environments that matter most are those where success is hard to define, trajectories are long, tools are central, and failure modes are subtle. Coding and computer use sit squarely in this category, with coding as the holy grail.

Coding is not just another task—it is a meta-domain. It combines long-horizon reasoning, tool invocation, stateful context, error recovery, and verifiable outcomes. Crucially, it also offers dense feedback: tests pass or fail, programs compile or break, diffs can be evaluated, and performance can be measured. This makes coding one of the few domains where high-quality reinforcement signals are available at scale, even as the task itself remains cognitively demanding.

This is not just a theoretical advantage; it is already reflected in market demand. Coding is the largest application vertical for the leading AI labs. Claude Code reached roughly $1 billion in ARR within six months. Microsoft Copilot is already a multi-billion-dollar business, and OpenAI’s Codex is generating hundreds of millions in annualized revenue. These products sit at the frontier of long-horizon, tool-heavy agent behavior and are natural early consumers of RL environments.

The most successful AI-native applications—and the likely first enterprise adopters of RL environments to stay competitive—are overwhelmingly coding-centric. Cursor is approaching $2B in ARR, with teams like Windsurf/Cognition close behind, and a fast-growing cohort of newer startups such as Cline and Kilo AI emerging. These companies are not experimenting at the edges; they are pushing agents into production workflows where reliability, verification, and continuous improvement matter.

By contrast, many RL environment companies are adopting environment-first strategies optimizing for surface area rather than depth. Teams build dozens of narrow environments—CRM editing, scheduling / email drafting, Slack clones—that demonstrate capability but rarely compound. These environments are short-horizon, weakly coupled to tools, and often rely on brittle heuristics or human-in-the-loop grading. They are easy to demo, but hard to generalize, and they produce limited reusable infrastructure. They provide initial value to labs training computer use, but will quickly commoditize.

Depth compounds in ways breadth does not. Teams that invest deeply in coding and complex computer-use environments are forced to solve hard, reusable problems early: scalable replication training, hybrid verification, trajectory replay, and eval-to-performance correlation. Improvements in these systems transfer across tasks within the domain and, over time, into adjacent domains like data analysis, DevOps, and general computer use.

This is why coding and computer use are the fastest initial pickup markets for RL environments. They sit at the intersection of high economic value, high task complexity, and strong verifiability. Teams that win here are not just shipping better agents—they are building the core training and evaluation infrastructure that makes broader generalization possible later.

Key Takeaway: Early advantage in RL environments accrues to teams that go deep in a small number of complex, high-signal domains—especially coding and computer use. Breadth can be added later. Depth, once skipped, is hard to recover. Coding is the single best area of expertise to build a foundation off of.

Market Landscape Today in 2026

The emergence of RL environments doesn’t displace incumbent labeling vendors, but it reshapes where value accrues. Traditional labeling is roughly a $5B market today (growing in excess of 50% YoY). The broader AI training data market ahead is likely to be significantly larger—but structurally different. Scale and workforce orchestration mattered in the labeling cycle. In the next one, research depth, domain specialization, and systems-level integration matter far more.

Incumbents like Surge AI, Mercor, Turing, Invisible, and others already generate billions in aggregate revenue serving frontier labs with expert labeling, fine-tuning data, and evaluations. That demand is expanding, not shrinking—especially in coding, which is already the highest-value expert vertical across these platforms. Coding-heavy expert work is where long-horizon reasoning, tool use, and verifiability converge, making it the natural bridge from labeling to RL environments.

That said, most incumbents are still optimized for a services-first, episodic workflow model, not for continuous learning systems. They aren’t research-first orgs at their core today. RL environments require reusable environment abstractions, deterministic task design, scalable replication training, and tight eval-to-performance feedback loops. These are infrastructure and research problems, not labor problems. The founding makeup of teams like Bespoke and Applied Compute are a mix of ML Scientists/Engineers and FDEs, not labelers, project managers, and software engineers.

It would not be surprising if one or two incumbents with strong research organizations successfully adapt to this new vertical. Surge AI is the clearest candidate today, and Snorkel has the intellectual foundations even if it lacks scale. But structurally, this market favors specialists built natively for RL environments. Depth compounds faster than breadth, and teams designed from day one to build training systems—not just manage expert throughput—are best positioned to capture the long-term value.

Which companies will win in 2030?

By 2030, the winners in RL environments will not be the teams with the most demos, the widest environment catalogs, or the largest services organizations. They will be the teams that build real infrastructure—and do so in close partnership with frontier labs to industrialize replication training, verification, and long-horizon environment orchestration.

That reality narrows the field. There will likely be 3 to 5 significant winners as labs concentrate spend among a small set of trusted partners. This market may end up resembling data labeling, which already has roughly three $1B+ revenue players (Scale, Mercor, Surge) and more than five $100M+ players (Turing, Micro1, Invisible, Handshake, etc.).

Most early-stage players lack one of the two non-negotiables: deep research capability or sustained lab trust. Without both, teams risk stalling as forward-deployed shops or commoditized environment builders as labs consolidate spend with a small number of thought partners. Incumbent labeling vendors will participate in this market, but few are structurally positioned to lead it.

In the end, this market will not be won by teams that apply RL environments fastest, but by those that help labs define how RL environments should exist at all. The enduring platforms will be built by teams that partner with frontier labs to turn ad-hoc experimentation into replication training systems, fragile graders into robust verification loops, and one-off demos into durable infrastructure.

That work is slow, research-heavy, and often invisible from the outside, but it compounds. And by the time RL environments become an obvious, enterprise-scale category, the winners will already be decided: not by market share today, but by who the labs trusted to build the training layer when it still mattered most.

As AI inference moves from experimentation to production, the infrastructure layer is no longer converging on a single “best” model. Instead, it is bifurcating into two durable and economically attractive markets, each optimized for a different constraint.

On one side are reserved and hourly compute platforms, designed to deliver predictability, control, and determinism for customers willing to operate infrastructure directly.

On the other are inference APIs, which trade some control for scale, absorbing utilization risk and abstracting complexity in exchange for cost efficiency and speed.

Understanding this split—and the tradeoffs it implies around utilization, performance, and deployment—is essential to evaluating where value will accrue as inference becomes ubiquitous.

Reserved Compute Platforms vs Inference APIs: Two Attractive Markets with Different Strengths

The AI infrastructure market is bifurcating into two durable and attractive segments, each optimized for different customer priorities and economic tradeoffs:

• Reserved / hourly compute platforms optimize for predictability and control

• Inference APIs optimize for utilization efficiency and cost abstraction

Neither is categorically superior; each wins in a different region of the workload and buyer space.

1) Reserved / Hourly Compute Platforms: Predictability and Control

Compute platforms offering hourly or reserved GPU instances (e.g., SF Compute, Modal–style offerings) are attractive because they deliver determinism.

What customers are buying

• Guaranteed access to scarce GPUs

• Predictable performance and capacity

• Full control over the runtime and stack

• Clear, stable unit economics (GPU-hours)

Why this is valuable

• Many workloads require consistency more than elasticity

• Teams want to size infrastructure once and operate against it

• Enterprises value control, compliance, and reproducibility

Economic profile

• High utilization per customer when workloads are well sized

• Revenue tied to long-lived contracts or reservations

• Smaller number of customers, but:

  • Larger ACVs

  • Lower churn

  • Clear capacity planning

Tradeoff

• Limited aggregation across customers

• Growth scales linearly with GPU fleet expansion

• TAM constrained by customers willing to operate infrastructure

This model resembles:

Early AWS EC2 reserved instances, CoreWeave-style compute, or traditional hosting—capital intensive, but predictable.

2) Inference APIs: Utilization, Cost Abstraction, and Scale

Inference APIs are attractive for the opposite reason: they absorb complexity and utilization risk on behalf of customers.

What customers are buying

• Tokens or requests, not hardware

• Elastic capacity without provisioning

• Built-in batching, routing, and scaling

• Operational simplicity and fast time-to-production

Why this is valuable

• Most inference workloads are bursty and uneven

• Most teams do not want to manage GPU fleets

• Cost predictability at the application level matters more than raw unit cost

Economic profile

• Lower utilization per customer

• Higher utilization at the platform level via aggregation

• Revenue scales with:

  • Number of customers

  • Number of use cases per customer

  • Overall inference volume

Tradeoff

• Less deterministic per-request performance

• More exposure to price compression at the execution layer

• Requires sophisticated systems to maintain margins

This model resembles:

Stripe, Twilio, or Datadog—abstracting infrastructure into a product.

3) The Core Tradeoff: Predictability vs Aggregation

Key insight

Reserved compute platforms win by making one customer predictable.

Inference APIs win by making many customers efficient together.

4) Why Both Markets Remain Attractive Long-Term

• AI workloads are diversifying, not converging

• Some buyers will always prioritize control and determinism

• Others will always prioritize speed, elasticity, and simplicity

As a result:

• Reserved compute captures value where predictability matters

• Inference APIs capture value where utilization and abstraction matter

Growth in one does not cannibalize the other; it often expands the total market.

Clean takeaway

Reserved compute platforms and inference APIs are both attractive, but for opposite reasons: one monetizes predictability with fewer, larger customers; the other monetizes utilization through aggregation across many smaller customers.

Inference Optimization

Utilization vs. Systems-Level Optimization: Why Utilization Dominates Inference Economics

While numerous systems-level optimization companies have developed techniques to enhance performance, only inference APIs—which directly manage GPU resources—possess the capability to optimize utilization.

A common intuition in inference infrastructure is that systems-level optimizations—higher tokens/sec, better kernels, smarter schedulers—are the primary drivers of durable advantage. In practice, for inference APIs, utilization is the first-order determinant of cost, margin, and competitiveness, while systems-level optimizations are second-order and increasingly commoditized.

1) The Two Levers Are Fundamentally Asymmetric

There are two ways to lower effective cost per token:

1. Increase tokens per second (performance)

2. Increase utilization (time GPUs are busy)

These levers behave very differently.

• Performance improvements reduce cost only if demand exists to consume the additional throughput.

• Utilization improvements reduce cost unconditionally, by eliminating idle time.

For most inference workloads—especially user-facing, bursty workloads—idle time dominates waste, not insufficient throughput.

2) Performance Advantages Compress Because Techniques Are Open and Diffuse

(With Representative Vendors)

Across modern inference APIs running similar model classes (e.g., 70B-parameter LLMs on A100/H100-class hardware), real-world tokens/sec variance is relatively narrow and shrinking.

Observed production ranges typically look like:

• Lower-throughput, aggregation-optimized APIs:
  ~120–150 tokens/sec

• Higher-throughput, latency-optimized APIs:
  ~220–260 tokens/sec

This ~1.3–2.1× spread reflects different optimization priorities rather than fundamentally different technology.

Representative Vendors by Throughput Orientation

(Illustrative, not exhaustive; varies by model, region, and configuration)

Higher Tokens/sec (Latency / Performance Optimized)

These platforms tend to push hardest on kernel-level and model-specific optimizations, often trading off breadth or flexibility.

• Fireworks AI

  • Heavy focus on latency and throughput (tokens/s)

  • Aggressive batching and kernel tuning

  • Narrower operational envelope per model

• BaseTen (dedicated / reserved setups)

  • High sustained throughput when utilization is high

  • Performance stability prioritized over elasticity

  • Best-case tokens/sec when capacity is fully consumed

• Select hyperscaler inference offerings

  • Deep integration with hardware-specific runtimes

  • Strong performance under controlled workloads

Lower Tokens/sec (Aggregation / Cost Optimized)

These platforms optimize for heterogeneous demand, model breadth, and utilization, rather than peak throughput.

• DeepInfra

  • Broad model catalog

  • Conservative batching to support mixed workloads

  • Optimized for platform-level utilization rather than per-model speed

• Together.ai (non-reserved, general API paths)

  • Trades some peak throughput for flexibility and rapid model availability

  • Performance varies widely by model and load

• General multi-tenant inference APIs

  • Support many models and traffic shapes concurrently

  • Throughput traded off in favor of reliability and aggregation

Why These Differences Persist—but Compress

The key observation is not who is faster, but why the gap is bounded:

• The fastest vendors rely on:

  • Flash-Attention variants

  • TensorRT-LLM

  • Triton kernels

  • KV-cache and batching strategies

• These techniques are:

  • Largely open source

  • Rapidly upstreamed into shared libraries

  • Adopted across vendors within quarters

As a result:

• Performance leaders change frequently

• The slowest credible APIs rarely fall more than ~2× behind

• The economic impact of performance differences is often dominated by utilization

Key Implication

Being the fastest inference API confers temporary advantage; being the most utilized confers durable advantage.

3) Utilization Variance Is Larger and More Durable

By contrast, utilization variance across inference platforms is both larger and more persistent.

It is common to observe:

• ~40–55% utilization on lightly aggregated or partially reserved systems

• ~70–85% utilization on well-designed, multi-tenant inference APIs

This represents a ~1.5–2.0× difference in effective cost, independent of tokens/sec.

Crucially:

• Utilization is driven by aggregation, customer mix, traffic heterogeneity, admission control, and product abstraction.

• These are business-model and architecture choices, not kernel tweaks.

• They do not diffuse via GitHub.

4) Concrete Comparison: Highest vs Lowest Tokens/sec Inference APIs

To make the tradeoff tangible, compare the best-case performance leader with a conservative, aggregation-optimized inference API running the same model class under production conditions.

For a 70B-parameter LLM on A100/H100-class hardware:

• Highest-tokens/sec inference API: ~230–260 tokens/sec

  • Aggressive kernel fusion

  • Highly tuned attention implementations

  • Large batch sizes under load

  • Often prioritizes throughput/latency over flexibility

• Lowest-tokens/sec (credible) inference API: ~120–150 tokens/sec

  • Conservative batching

  • Broader model catalog

  • More heterogeneous workloads

  • Optimized for reliability and aggregation

This is roughly a ~1.7–2.1× gap in raw tokens/sec.

Now layer in utilization:

Despite being nearly 2× slower on paper, the aggregation-optimized platform can deliver meaningfully better economics because its GPUs are busy far more of the time.

Two observations follow:

1. The fastest API is rarely the cheapest in practice. Peak tokens/sec only matters when demand is perfectly aligned and sustained—uncommon outside narrow workloads.

2. The performance gap is bounded and compressing. Even the slowest serious inference APIs are usually within ~2× of the fastest, while utilization differences regularly exceed that.

In inference APIs, the difference between “best” and “worst” performance is bounded; the difference between “busy” and “idle” infrastructure is decisive.

5) When Systems-Level Optimization Still Matters (Second-Order)

This does not imply systems work is unimportant.

Performance optimizations matter most when:

• Utilization is already high and stable

• Latency directly impacts revenue

• Workloads are homogeneous and predictable

• SLAs are strict

In these regimes, performance improvements compound on top of utilization.

However, for most inference APIs, performance is a multiplier on utilization, not a substitute for it.

Clean takeaway

Utilization variance dominates performance variance—and unlike performance optimizations, utilization advantages do not open-source away.

Strategic Synthesis

Utilization determines whether an inference API is competitive at all.

Systems-level optimizations determine how competitive it is once it is.

Because performance techniques are largely open, shared, and fast-diffusing, durable advantage in inference APIs comes from aggregation, abstraction, and control, not raw execution speed.

Utilization Assumptions (for Pricing & Cost Models)

All cost and unit-economics comparisons in this report assume observed production utilization, not theoretical peak throughput. A critical distinction throughout is between customer-level utilization and platform-level utilization—they behave very differently and are often conflated.

Key Definitions

• Customer-level utilization: How busy GPUs are for a single customer or single workload.

• Platform-level utilization: How busy GPUs are across the entire platform, aggregating many customers and workloads.

Most misunderstandings come from applying customer-level intuition to platform-level economics.

Utilization Tiers Used Throughout

10–40% Utilization: Customer-Level Reality (Small, Isolated, or Reserved)

This band is normal for individual customers, even when systems are well engineered.

Who lives here

• Series A–C AI startups
  Enterprise internal AI teams

• Teams using reserved GPUs or BYOC deployments

• Products with one or two core AI features

Why utilization falls in this range

• Capacity is provisioned for p95–p99 traffic, not averages

• Demand is bursty and diurnal
  Multiple models or pipelines fragment GPU memory

• Latency SLOs prioritize headroom over cost efficiency

Representative examples

• A Series B startup running reserved A100s for a customer-facing copilot

• A Fortune 500 company deploying an internal HR or IT chatbot

• A SaaS product with one AI feature used heavily during business hours

Key point

Utilization in the 10–40% range reflects traffic isolation and risk tolerance, not poor execution.

This is the dominant reality for most companies below platform scale.

40–60% Utilization: Mature Single-Tenant or Light Aggregation

This band reflects operational maturity without hyperscale aggregation.

Who lives here

• Mid-size SaaS companies with steady AI demand

• Large enterprises with a single, heavily used internal platform

• Vendors offering semi-dedicated capacity to a small number of customers

Why utilization improves

• Demand becomes more predictable

• Fewer models are active per GPU
  Batching and admission control are consistently applied

Representative examples

• A public AI SaaS company with steady weekday traffic

• A large enterprise analytics copilot used company-wide

Important constraint

• This is often the upper bound for single-tenant or lightly multi-tenant systems

• Pushing beyond ~60% usually requires:

  • Aggregating many customers, or

  • Accepting higher tail latency

60–85% Utilization: Platform-Level Aggregation (Large Platforms)

This band is structurally limited to platforms that aggregate many customers.

Who lives here

• Multi-tenant inference APIs

• Large inference platforms serving many startups

• Hyperscaler-managed inference services

Why utilization is high

• Many small, asynchronous demand curves smooth out

• Peaks rarely align (statistical multiplexing)

• Platforms can batch aggressively and queue intelligently

• Startup customers tolerate shared infrastructure and mild latency variance

Representative examples

• OpenAI-style public APIs

• Large inference platforms serving hundreds or thousands of startups

Critical caveat

• Sustained utilization above ~85% sharply increases tail latency and failure risk

• Most user-facing platforms deliberately operate below this ceiling

Implications for the Rest of This Report

1. Startup-heavy platforms can still achieve high utilization
   Individual startup customers are low-to-moderate utilization, but aggregating many of them drives platform-level utilization into the 60–85% range.

2. Reserved and BYOC models structurally reduce utilization
   Pinning capacity to customers collapses the benefits of aggregation.

3. Utilization differences dominate cost outcomes
   A platform running at 70–80% utilization will typically outperform a faster system running at 40–50%, even if tokens/sec is lower.

Utilization is primarily a function of aggregation and abstraction—not company size or engineering quality.

All subsequent cost, performance, and market analysis should be interpreted through this lens.

Why 10–25% Utilization Is Real (and Common)

When people hear “10–25% GPU utilization,” they often imagine mismanagement. In practice, it usually reflects normal product behavior, not incompetence.

1) Early-Stage and Growth-Stage Products

For most inference-backed products before real scale:

• Traffic is bursty (peaks during business hours, troughs at night)

• Feature usage is uneven (one agent or endpoint dominates)

• Capacity is sized for p95 or p99, not average

Result:

Even “healthy” systems sit at 10–30% average utilization.

This mirrors:

• Web servers pre-autoscaling

• Databases sized for peak QPS

• Early SaaS infra economics in general

2) Reserved Capacity Is Sized for Worst Case, Not Average

Any system with:

• SLAs

• Latency SLOs

• Human-facing UX

must provision for spikes, not medians.

Example:

• Peak demand: 5× average

• You size for peak

• Average utilization → ~20%

This is by design, not error.

3) Multi-Model and Multi-Tenant Fragmentation

Utilization collapses when:

• Multiple models are deployed in parallel

• Each has independent traffic patterns

• GPU memory constraints prevent consolidation

Even if total demand is high:

• Individual GPUs may be underutilized

• Fragmentation blocks efficient packing

This is especially common in:

• Enterprise copilots

• R&D-heavy teams

• Products experimenting with multiple models

4) Why Serverless APIs See 60–85% While Customers See 10–25%

This is the key contrast.

Single customer / reserved setup

• Demand curve = narrow

• Variance = high

• Utilization = low

Aggregated serverless platform

• Demand curves from thousands of customers overlap

• Variance smooths out

• Utilization rises to 60–85%

This is not magic — it’s statistical multiplexing.

Is 10–25% the Floor or the Average?

Important nuance:

• 10–25% is common for:

  • Early-stage products

  • Reserved GPUs

  • Internal tools

• 40–60% is achievable for:

  • Mature SaaS with predictable usage

  • Well-tuned admission control

• 70–85% is rare outside:

  • Large serverless platforms

  • Highly constrained, always-on workloads

Sustained >85% utilization is dangerous unless you’re:

• Willing to drop requests

• Willing to increase latency

Quantitative Reality Check

Consider a product with:

• 8 hours/day of peak usage

• 16 hours/day of low usage

• 2× peak-to-average ratio

Even with good smoothing:

That’s a well-run system.

Now add:

• Model experimentation

• Geographic skew

• Release cycles

You’re back in the 20–30% range quickly.

Why This Matters Strategically

This is why:

• Reserved GPU inference looks cheap on paper and expensive in reality

• Performance optimizations often disappoint economically in isolation

• API-based inference dominates long-term share

It also explains why:

• Systems that maximize utilization outperform those that maximize peak throughput

• Control planes beat raw execution engines

Inference APIs vs. Customer Utilization: Why Startup-Heavy Platforms Can Run High Utilization

A common misunderstanding in inference economics is the assumption that serving mostly startups implies low utilization. This confusion comes from conflating customer-level utilization with platform-level utilization—two very different things.

Customer-Level vs Platform-Level Utilization

At the customer level (typical startup behavior):

• Utilization is often low (10–30%)

• Traffic is bursty and uneven

• Workloads are small and episodic

• Capacity is sized for peaks, not averages
  

This is true for most startups, and it is entirely rational.

At the platform level (inference APIs):

• Hundreds or thousands of independent demand curves are aggregated

• Peak usage times rarely align

• Idle capacity smooths out statistically

This is classic statistical multiplexing.

A platform serving many low-utilization customers can itself operate at 60–85% utilization.

This is not unique to inference. It is exactly how infrastructure platforms like:

• AWS EC2

• Datadog

• Stripe

• Twilio

achieved strong unit economics early—despite serving predominantly small customers.

Why Startup Customers Often Improve Platform Utilization

Counterintuitively, startup-heavy customer bases are often better for utilization than a few large customers.

1) Asynchronous, Heterogeneous Workloads

Startups tend to:

• Operate across time zones

• Have different peak hours

• Use different models, endpoints, and request shapes

This creates natural demand smoothing.

Contrast this with:

• One large enterprise customer

• One internal copilot

• One dominant model

Those scenarios concentrate demand and make it harder to keep GPUs busy.

2) Fewer Hard SLAs

Startup customers are typically more tolerant of:

• Mild latency variance

• Queueing under load

• Shared infrastructure

That tolerance allows inference APIs to:

• Batch more aggressively

• Apply stricter admission control

• Run GPUs closer to saturation

By contrast, enterprise customers often impose:

• Dedicated capacity requirements

• Strict tail-latency guarantees

• Isolation constraints

All of which structurally reduce utilization.

3) Minimal Demand for Reservation or Isolation

Most startups:

• Do not request dedicated GPUs

• Do not require VPC isolation

• Default to fully serverless APIs

This allows inference platforms to:

• Pool capacity globally

• Move workloads dynamically

• Optimize scheduling and batching across customers

That flexibility is the core utilization advantage of inference APIs.

When Utilization Drops for Inference Platforms

There are two scenarios where even inference APIs experience utilization pressure.

1) Growth in Reserved or Dedicated Capacity

If a platform:

• Pins GPUs to specific customers

• Sells large amounts of reserved capacity

• Guarantees isolation or fixed latency

Then utilization begins to resemble:

• 40–60% in well-sized cases

• 20–40% when demand variance is high

This is why reserved inference tends to be expansion revenue, not the economic core.

2) Heavy Skew to a Single Model or Workload

Utilization also drops when:

• A single model dominates demand

• Requests cluster in time

• Latency SLOs are strict

In these cases:

• Capacity must be overprovisioned

• Scheduling flexibility decreases

However, these are also situations where:

• Pricing power increases, or

• Routing and scheduling logic becomes strategically valuable

Where Inference APIs (i.e. DeepInfra, Fireworks, Together) Sit (Illustrative, Not Unique)

For a modern inference API with:

• A serverless model

• A startup-heavy customer base

• A broad catalog of models

It is reasonable to expect:

• Platform-level utilization: ~60–80%

• Customer-level utilization: often 10–25%

• Effective economics: closer to hyperscale APIs than reserved-GPU platforms

In other words:

Startup customers are low-utilization individually.

Inference APIs that aggregate them do not have to be.

This dynamic is inherent to the API model itself, not specific to any single provider.

Why This Matters Strategically

This distinction explains several broader conclusions in the report:

• What segments of the market API-based inference versus reserved capacity captures long-term

• Why utilization variance dominates performance and tokens/sec variance

• Why inference APIs are structurally different from reserved-GPU platforms

It also clarifies why inference APIs optimized for aggregation and cost discipline are not simply “worse versions” of platforms optimized for model novelty or research adjacency. They are solving different problems and, critically, exhibit different utilization dynamics.

Enterprise infrastructure is entering a bifurcated cycle: an immediate scramble to operationalize AI under extreme resource constraints (Act 1), followed by a structural rewrite of organizational architecture itself (Act 2). The firms that win the next decade will bridge these worlds — capturing momentum now while accumulating long-duration advantage as AI becomes the primary consumer and producer of work.The evolution of enterprise infrastructure can be viewed as a two-act play.

Below are the key needs and constraints defining each act.

Act 1:

• Supply-side constraints (GPU scarcity, infra fragmentation).

• Tools that accelerate building (acceleration, security, orchestration).

• Many will be temporary arbitrage plays; some will become rails.

Act 2:

• A structural rewrite of “how work happens.”

• Systems treat agents as organizational primitives.

• Entire SoRs (AD, HRIS, ERP, IdPs, ticketing) get replaced.

Act 1 is the AI-native momentum phase: a period of acute infrastructure bottlenecks, new AI fabrics, and a proliferation of tools for not just building/securing but accelerating/unblocking AI-native applications and processes. Many of these bets may likely not endure into meaningful companies but could be good investment outcomes nonetheless.

Act 2 represents the structural transformation phase: where firms that dominate Act 2 will emerge not from “micro” rewrites using AI, or tooling where the foundational layers can eat into over time, but from foundational redefinitions of “how work happens” AND “lessons learned” from Act 1 that help create more enduring systems — imagine an enterprise org hierarchy not just being defined by employees but also agents as first class organizational elements - entire systems of record such as HR and core infrastructure services such as Active Directory will need to be reimagined.

So the strategic opportunity for us lies in balancing these acts — making targeted investments across both acts, balancing near-term momentum with long-term structural vision. This involves a 3 pronged approach:

1. Act 1 Investments: These investments will focus on the “tracks” of the current AI build out. Opportunities in this category are driven by immediate, intense demand and clear market dislocations, such as the ongoing GPU shortage. The key discipline here will be to differentiate between durable infrastructure plays and temporary arbitrage opportunities that will fade as the market matures.

2. Act 2 Investments: These investments will target “category creation” opportunities that fundamentally rewrite existing software and security paradigms. These are higher-risk, longer-horizon ventures that require a deep conviction in a company’s vision to see beyond the limitations of the current stack. (eg ..?)

3. The Bridge (Act 1+2): We also believe there will be valuable investments in companies whose Act 1 products are necessary to solve today’s pressing problems but are architected to become the system of record for the Act 2 paradigm. These are the platforms on which we will “double down,” as they offer the potential for both near-term revenue and long-term, defensible market leadership (eg Theom to not just replace current data classification, data lake protection solutions but also as a foundational layer for enterprises to rely for on a data governance platform to unlock agentic deployments).

   

II. Looking ahead to Act 2 structural transformation and working backwards into Act 1 and Bridge Opportunities

Some themes are proposed in the visual for discussion with some details on a few below:

Democratization of AI Clouds

Today’s compute economy is defined by scarcity — the “GPU-constrained” world.

Neo-clouds (CoreWeave, Nebius, Lambda Labs, Crusoe, etc.) and cloud fabrics that stitch across hyperscalers have emerged to address this shortage. These players capture short-term, ?-margin value in solving supply-side inefficiency while setting the stage for longer-term abstraction layers. We first wrote about this phenomenon in 2023 with “The Financialization of the Data Center” piece. Some will make it to Act 2, others won’t.

The next step will come from eventual democratization of GPU supply (e.g. multiple GPU makers, valid non-GPU options such as TPUs becoming mainstream etc), but also from virtualization and workload orchestration that lets developers treat heterogeneous compute as fungible while also serving local markets with a latency sensitive orientation. Sovereignty concerns will also play into the possible democratization of “country specific” GPU clouds.

We also believe future acts will be defined by heterogenous compute — across other types of GPUs, TPUs, ASICs and the CPU. This shift will have massive derivative effects across enterprise infrastructure.

LLMs ⇒ World Models

This is currently a deeply discussed and contested discussion in the AI community. Current LLMs possess a remarkable ability to process and generate text, but they lack a true, underlying understanding of cause and effect. The next frontier in AI research is the development of “World Models.” A world model is an internal simulation of an environment that allows an agent to predict the consequences of its actions before taking them and builds upon the mathematical underpinnings of Markov Decision Processes (MDPs) which is a model sequential decision-making in situations with uncertain outcomes. Formally, it learns the transition function of the environment, or the probability of ending in a new state s’ given the current state s and an action a, expressed as P(s’|s,a).

Recent research has shown that today’s LLMs can be “induced” to perform basic world model functions, such as determining the preconditions for an action and predicting the resulting state change. This is a foundational step toward creating agents that can truly reason, plan, and operate in complex, dynamic environments.

Some example areas of interest include:

• Advanced Simulation Environments: Platforms capable of generating vast quantities of high-fidelity synthetic data, which will be essential for training models to understand complex cause-and-effect relationships in a controlled manner. Simulation-first companies include Waabi, Applied Intuition etc and Synthetic Data Platform companies include Synthesis AI, Datagen, Sky Engine AI etc.

• Data Platforms for Multimodal Reasoning: The real world is not just text. World models will need to ingest and correlate information from a wide array of sources, including video streams, sensor data from IoT devices, and structured logs from enterprise systems. Investing in the data platforms that can unify and process this multimodal data is a tangible way to invest in the future of world models. Categories here include Physical Video and Sensor platforms

Robotics/IoT data platforms with companies like Viam, Formant etc. Even a next gen storage can be envisioned here for storage platforms purpose built for large scale simulations and multimodal training.

The real enterprise unlock is not synthetic environments — it’s stable, predictable agent behavior. World models are the missing reliability layer.

Agents ⇒ Things

Act 2 extends the agentic wave into the physical domain — robots, sensors, and “things” with agency. This transformation moves AI from desktops and servers into real-world decision loops — factories, logistics, and healthcare. Emerging players like Sanctuary AI, Figure, and Covariant represent the early prototypes of this convergence. These “things” may require a new hardware stack to enable them and the thinking will likely be split between on device and in the cloud.

• Edge inference + local autonomy ← critical for sovereignty + latency

• A hybrid stack: on-device reflexes + cloud-level planning

Act 2 physical systems will adopt a layered control stack: reflexive inference at the edge, and world-model-driven planning in the cloud.

Micro ⇒ Chunky SaaS

Workflows themselves will be rewritten from first principles. Instead of AI co-pilots within human-centric systems, Act 2 SaaS will be AI-first orchestration platforms that treat humans as optional participants in the loop. This could produce AI-native ERP, finance, and HR systems — startups like Doss and Quanta hint at this future.

Chunky SaaS collapses dozens of micro workflows into autonomous processes that produce outputs, not dashboards. If Act 1 gave us copilots, Act 2 gives us “self-driving back-office systems.”

III. The Bridge - areas where Act 1 monetization builds into Act 2 defensibility:

We discuss at depth the Agentic Cyber and Agentic Lifecycle themes in other pieces.

A brief overview of the AI Networks theme is included below (a detailed introduction into this topic is here.

• Why: Almost all clouds are at <50% GPU utilization with the primary bottleneck for scaling AI being the network. Traditional networking was built for statistical multiplexing (many small, independent jobs), but large-scale AI training operates as a single, massive, synchronized job across tens of thousands of accelerators, demanding all-to-all communication where the worst-case latency dictates job completion time.

• What: The solution is to evolve from a reactive, “best-effort” network to a proactive, deterministic, and “perfectly scheduled” fabric. The goal is to make the network as predictable as an on-chip interconnect, removing random delays and jitter that stall expensive accelerators.

• How: This is achieved through a co-design of hardware and software including next gen data center components such as Attotude, Enfabrica and discrete “sub systems” that offload transport and congestion control from the CPU, nanosecond-level time synchronization across the data center, allowing the network to precisely schedule every single data packet and ensuring a predictable, ultra-low-latency, and near-flawless fabric

V. Summary

We believe some of the most successful investments will capture Act 1 economics while accumulating Act 2 optionality.

The winners will exhibit a unique trait: Act 1 capital efficiency, Act 2 architectural inevitability.

Below you can see an example described by Sierra:

Why the ADLC Exists

The Agent Development Lifecycle (ADLC) defines how autonomous systems move from prototype to production, establishing the reliability and governance foundation for the agentic era. Just as the Software Development Lifecycle (SDLC) industrialized software engineering, the ADLC is doing the same for intelligent agents.

Below are a few key areas that are emerging as key parts of the ADLC stack:

The central insight: enterprises will pay for reliability, control, and visibility over autonomy—not for intelligence alone.

The last wave of agent frameworks proved agents could plan and act but failed to deliver operational reliability. They were stateless, brittle, and non-deterministic, unable to meet enterprise standards for safety, compliance, and interpretability. The next generation—anchored by Databricks Agent Bricks, OpenAI Agents SDK, and Temporal—brings new foundational infra for durability and control.

Winning companies will:

1. Abstract complexity into deterministic, explainable primitives.

2. Integrate with enterprise control planes for identity, policy, and observability.

3. Monetize reliability, not model intelligence.

The ADLC represents the industrialization of AI autonomy: a shift from building agents that can think to building infrastructure that ensures they can be trusted.

Side note: Greg seems to have adopted Jensen Huang’s signature Tom Ford leather jacket look for the announcement. The leather jackets are a signature look for Jensen as he reportedly owns over 50 of them and wears them almost exclusively at public events. The look has evolved into a brand symbol—so much so that analysts often call NVIDIA launches “the leather jacket show.” Jensen seems to appreciate strong American engineering and have to admit, it’s a sharp look!

Scale check

• 10 GW ≈ 4–5M GPUs (depending on racks and efficiency). That’s effectively Nvidia’s entire 2025 production and nearly double last year’s shipments.

• When Jensen Huang called the deal “monumental,” it wasn’t marketing—it was literally about the industrial scale of the project.

Financing geometry

• Nvidia invests $10B in equity for every 1 GW OpenAI deploys, up to $100B at 10 GW.

• For every ~$35B of GPU orders, Nvidia chips in ~$10B of its own stock. Net effect: OpenAI pays ~71% in cash, ~29% in equity.

• In plain terms, Nvidia is partially financing its own sales—a structure that ensures demand while tying OpenAI closer to its roadmap.

Breaking from Microsoft

This scale-up also highlights a deeper shift: OpenAI loosening its dependency on Microsoft.

• Microsoft pushed hard to block OpenAI’s Windsurf acquisition, worried it would threaten GitHub Copilot’s moat.

• The partnership has grown tense: Microsoft wants OpenAI locked into Azure, while OpenAI wants flexibility to chase cheaper, more abundant capacity.

• By cutting deals with Oracle for cloud and Nvidia for capital + silicon, OpenAI is deliberately hedging away from Microsoft’s grip.

The “enemy of my enemy” logic

Nvidia is equally motivated and pulling a Michael Corleone to seize power. The big three clouds—Amazon, Google, Microsoft—are all investing in their own silicon to escape Nvidia’s margins. So Nvidia has gone the opposite way:

• Prop up challenger clouds like Oracle, CoreWeave, Nebius, Crusoe.

• Seed alternative capacity pools that keep demand diversified and hyperscaler leverage in check.

• In that light, Nvidia and OpenAI’s alignment is classic: the enemy of my enemy is my friend.

The capital loop

The money flows almost like a cartoon: OpenAI spends with Oracle → Oracle buys from Nvidia → Nvidia invests into OpenAI → repeat. On paper it looks circular, even bubble-like—capital bouncing between the same three names.

But the distinction is that each spin leaves behind physical assets: GPUs shipped, racks powered up, substations wired, models deployed to hundreds of millions of users. It’s not just numbers recycling across balance sheets—it’s dollars hardening into infrastructure.

The historical rhyme

This isn’t tulips or banner ads. The better analogies are railroads in the 1800s or fiber buildouts in the early 2000s. Overcapacity wiped out equity holders in the short term, but the infrastructure remained and became foundational.

Risks

• Power access: interconnect queues and substations are the true bottleneck now, not chips.

• HBM supply: memory packaging could limit throughput regardless of how many GPUs are bought.

• Utilization: 10 GW of idling silicon would crater returns; keeping GPUs busy is existential.

• Policy: tariffs, permitting, or AI regulation could reshape the economics overnight.

Investor lens

• Watch LCOC (levelized cost of compute): who can turn megawatts into tokens cheapest?

• Vendor capital as moat: Nvidia financing its own demand gives it stickiness no startup chip vendor can replicate.

• Second-order bets: optics, HBM, advanced packaging, switchgear, immersion cooling—all benefit from this cycle no matter who “wins” the model race.

Bottom line

OpenAI’s 10 GW plan isn’t just capex—it’s a bid for independence from Microsoft, fueled by Nvidia’s need to weaken the hyperscaler triopoly. The capital loop may look like a meme, but it’s also a flywheel: capital → compute → models → demand → back to capital.

The question isn’t whether the cycle makes sense—it’s who will still be standing when the music slows.

dbt’s leadership is so strong that it has begun to commoditize the EL layer: if transformation is standardized in dbt, ingestion and loading can become utilities, not differentiators. That is a testament to the power of the dbt ecosystem.

dbt’s Lead: From Community to Compiler Intelligence

In January 2025, dbt Labs reinforced its technical moat by acquiring SDF Labs. The deal brought a Rust-based SQL compiler that enables dbt to truly understand SQL—not just execute it.

This acquisition cemented dbt’s role as the innovation engine in transformation, ensuring it remains central to the workflows of hundreds of thousands of analytics engineers.

Fivetran’s Move: Validating Transformation with Tobiko

Yesterday, September 3 2025, Fivetran announced the acquisition of Tobiko Data, the creators of SQLMesh and SQLGlot. SQLMesh and SQLGlot have already gained strong traction in the open-source community, and we highlighted them in our earlier Data Infrastructure in 2025 piece.

Fivetran, long known as the market leader in automated ingestion, now generates more than $300M ARR and is IPO-ready. By acquiring Tobiko, it signaled that owning transformation is essential to staying relevant in the modern stack—a direct validation of the standard dbt established.

What Tobiko adds for Fivetran:

• SQLMesh for incremental transformations and semantic awareness.

• Virtual Data Environments for safe, isolated development.

• SQLGlot for multi-dialect parsing across engines.

Tobiko’s founding team—Toby Mao (Airbnb/Netflix), Iaroslav Zeigerman (Netflix/Apple), and Tyson Mao—built SQLMesh based on years of experience with brittle pipelines at scale. Their vision of efficient, stateful transformations resonated with open-source users and Fivetran alike.

It’s important to note: this isn’t a zero-sum rivalry. dbt continues to lead transformation with unmatched community adoption shaping how data teams work. Fivetran’s acquisition of Tobiko is an acknowledgment—recognizing that transformation is the layer that truly matters, and that dbt defined the space.

Conclusion

Transformation is no longer the overlooked step between ingestion and dashboards. It is the center of the modern data stack, where governance, AI-readiness, and developer workflows converge.

• dbt Labs remains the anchor: the first name data teams reach for, the company pushing the boundaries with compiler intelligence and AI-assisted development.

• Fivetran is expanding its scope: Tobiko strengthens its offering, but more importantly, validates the centrality of dbt’s vision.

In the years ahead, dbt’s influence will only grow as AI and governance demand semantic understanding of data pipelines. Fivetran’s move underscores what the market already knew: transformation—pioneered and defined by dbt—is where the real value lies. Tobiko brings excellent technology to the Fivetran platform, but dbt continues to set the standard others follow.

Major congratulations to the Tobiko Data team on the validation—and credit to dbt Labs for creating the standard others are racing to follow

Source: Bloomberg and Census Bureau [link]

Intro: Data Centers Surpass Office Construction in 2025

This year, annual spending on new data centers is set to overtake general office construction. 10 years ago, US data construction was a paltry $2B compared to over $33B of office construction spend. After a decade of consistent growth up until COVID, office development has sharply declined from its 2020 peak, while data center investment has surged—driven by the AI arms race. This divergence underscores a profound shift in commercial real estate priorities: the physical backbone of the digital economy is becoming more valuable than the spaces where people work in person. AI’s insatiable appetite for compute, storage, and low-latency connectivity is reshaping the built environment at unprecedented speed.

1. Real Estate

The first key input to the build of the AI data center is real estate. The footprint of AI-driven data centers is scaling dramatically, from 275M sq ft in 2025 to 700M sq ft by 2030. The footprint of AI data centers looks more like a crypto mining data center than a traditional data center.

The real estate footprint of AI data centers is radically different than the traditional data center defined by Equinix and population density. Facilities are also getting physically larger, with hyperscale operators like Azure building campuses exceeding 2M sq ft—over 100× the size of a typical U.S. data center. AI data centers optimize for cheap land and cheap electricity, while traditional data centers optimized for bandwidth and proximity to population centers. CoreWeave’s 450K sq ft facilities illustrate the new “AI-native” build standard, optimized for high-density racks and liquid cooling. This expansion will demand favorable zoning, proximity to high-capacity fiber routes, and access to reliable power sources. The result: data center real estate is becoming one of the most competitive and strategic asset classes globally, representing more than 30% of Blackstone’s flagship fund. We wrote about this trend in 2023 and the financialization of the data center has only become stronger.

Notable facts:

• Blackstone now owns more data center square footage than some of the largest REITs, following its acquisitions of QTS and stakes in Digital Realty.

• DigitalBridge reports that AI tenant deals are often signed with prepayment or multi-year upfront commitments—rare in traditional colocation.

2. Capital

Data center development has become deeply financialized, with M&A volume hitting $73B in 2024. Private equity and infrastructure funds like Blackstone, KKR, and DigitalBridge are aggressively consolidating colocation providers, betting on multi-decade AI infrastructure demand. AI-native operators—CoreWeave, Crusoe, Lambda—are attracting both equity and debt financing to expand capacity quickly. Capital intensity is enormous: annual investment is projected to grow from $450B in 2025 to $1.2T by 2030. For investors, the blend of long-term leasing, mission-critical status, and AI-driven growth makes data centers a premier infrastructure play.

Notable facts:

• Private equity buyers are paying EV/EBITDA multiples in the mid-teens for hyperscale-ready assets, a premium over historical averages.

• AI-native operators like CoreWeave have secured multi-billion dollar credit facilities backed by GPU collateral, effectively making compute a financeable asset class.

3. Compute

The compute layer is entering an unprecedented growth phase, with chip shipments set to grow from 14M in 2025 to 70M by 2029. Only capital rich companies have the necessary compute to compete in AI, leading to a fragmentation of GPU rich and GPU poor companies. Aggregate AI compute capacity will surpass 200 exaflops by the end of the decade. Nvidia dominates the current revenue pool—GPUs represent the bulk of a projected $400B in AI chip sales by 2027—while TPUs, ASICs (i.e. by cloud providers), and AI-optimized XPUs (i.e. by Broadcom) also gain share. This growth is driving massive supply chain and manufacturing investments, from advanced packaging to high-bandwidth memory. In the AI era, data center competitiveness is increasingly a function of compute density per megawatt.

Notable facts:

• Nvidia’s current lead in AI accelerators gives it ~80% market share in data center GPUs, with gross margins exceeding 75%.

• HBM (High Bandwidth Memory) production is a key supply constraint—Samsung, SK Hynix, and Micron are all racing to expand capacity, with HBM prices up >30% YoY.

• By 2027, an AI training cluster capable of 1 exaflop may cost $500–$800M in hardware alone, not including networking or facilities.

• Arm-based CPUs are quietly gaining share in inference-heavy AI clusters due to lower power draw per token processed.

4. Energy

AI’s energy needs are accelerating at a scale that will reshape national grids. By 2030, AI workloads could consume 10% of total U.S. electricity, up from just 4% in 2023. Meeting this demand requires adding 90GW of capacity between 2025 and 2030—equivalent to building dozens of new nuclear plants or thousands of large-scale solar farms. AI power spending will nearly triple from $58B in 2025 to $191B by decade’s end. This dynamic makes energy sourcing and sustainability not just ESG talking points, but competitive differentiators in AI infrastructure.

• The AI power ramp—an extra 90GW by 2030—is roughly equal to adding the total current generation capacity of the United Kingdom.

• Data center PUE (Power Usage Effectiveness) averages ~1.2 for hyperscale, but AI training loads can push it higher due to extreme cooling needs.

• Some operators are exploring direct nuclear supply—Microsoft has hired nuclear engineers and filed for SMR (Small Modular Reactor) licenses.

• Curtailment agreements with renewable providers are being renegotiated to prioritize AI data center loads over less profitable grid exports.

5. Networking

High-performance networking is the circulatory system of AI data centers, with the total AI networking TAM projected to hit $100B by 2030. Leading players include Arista as the pure independent and NVIDIA, which has quietly grown its networking business to >$10B of sales after its acquisition of Mellanox. Optical transceivers alone will grow from a ~$12B market in 2024 to $40B by 2030, driven by the shift to 800Gb and above interconnects. The volume of 800Gb ports is expected to surge from 13M in 2025 to 284M by 2029, enabling both scale-out and scale-up AI clusters. Attotude’s low-latency interconnects, a $35B market by 2029, will be critical for tightly coupled GPU workloads. As models scale, the bottleneck shifts from compute to moving data fast enough between chips, nodes, and sites.

• The leap from 400Gb to 800Gb optical links can reduce model training time by double-digit percentages in tightly coupled GPU clusters.

• Co-packaged optics, still <5% of the market today, could surpass $15B TAM by 2030 as switch ASIC power budgets become limiting factors.

• The Infiniband vs. Ethernet battle is intensifying: Nvidia’s Quantum-2 Infiniband dominates training clusters, but cloud providers are pushing Ethernet-based RDMA to cut costs.

• Retimer and cable costs can account for >20% of the total networking bill in AI facilities—an often-overlooked capex driver. See our Astera Labs S1 review.

Enter the agent infrastructure stack.

If software was about deterministic instruction execution, and ML infra was about data pipelines and model tuning, agent infra is about managing stochastic, semi-autonomous systems that learn, act, and retry—without a human in the loop.

This is not an extension of existing infra. It’s a clean-slate redesign. Below are the five categories defining the stack.

🔧 1. Agent Builders: The New Interface Layer

Think LangChain, CrewAI, and Sema4. These frameworks abstract away the glue logic and let teams stitch models, tools, memory, and triggers into goal-driven agents.

Most enterprise prototypes today are built on glue-code sprawl: brittle chains of prompts, scripts, and Zapier-style logic. Agent builders aim to replace this with something composable, observable, and production-ready.

🧠 Strategic Insight: This layer will be hard to displace once embedded. Expect 2–3 dominant winners, each with distinct abstraction levels (think: Unix → Python → React).

⏱ 2. Persistence Engines: Durable Orchestration for Non-Determinism

Traditional workflow orchestrators (Airflow, Temporal) assume deterministic step execution. Agents violate that assumption. Their logic branches, retries, and re-plans on the fly.

That’s why companies like Zipline, Inngest and Hatchet are gaining traction. They act as a queue/cron + state store + retry engine—purpose-built for stochastic systems.

🧠 Strategic Insight: This category is inevitable. It will underpin any agent that runs longer than 15 seconds or involves a multi-step workflow. Bonus: strong data flywheel from agent execution traces.

🧠 3. Memory: Statefulness as Infrastructure

Current-gen agents are stateless. Each task begins with a fresh prompt. That’s broken. The move toward long-term memory will define whether agents feel brittle or adaptive.

Emerging vendors like Pinecone, Mem0 and Zep are trying to build structured memory layers—handling context, conflict resolution, summarization, and relevance—all with low latency.

🧠 Strategic Insight: Memory is both a UX unlock and an enterprise liability. Expect standards (e.g., memory protocols) to emerge for auditing, compression, and conflict resolution. This is a greenfield category but risks being swallowed by incumbents or embedded into agent frameworks.

🌐 4. Web & Tool Interaction: Killing RPA for Good

APIs are not enough. Most enterprise tools still rely on human GUIs. Agents need to control UIs programmatically. That’s the role of headless browsers + browser-native agent protocols like MCP.

Platforms like Browserbase let agents interact with SaaS interfaces as if they were humans. This is where RPA should have gone.

🧠 Strategic Insight: This is a linchpin for real-world integration, and it’s already splintering into protocol vs. execution layers. Whoever wins here will become the Zapier + Puppeteer for agents.

🔐 5. Agent Identity & Auth: No Enterprise Rollout Without This

Today’s agent security posture is terrifying. Tokens are hardcoded. Agents “borrow” user IDs. There’s no audit trail or permission boundary.

Agent-native auth systems like Composio and Arcade are building scoped, verifiable identities for agents—think service accounts + RBAC + session tracing + audit logs.

🧠 Strategic Insight: If you’re selling to a CISO, this is the entry point. Long-term, it becomes the enforcement layer for trust and compliance across the stack.

🧠 The Control Plane War Has Begun

Inference is now a solved problem—API pricing is dropping, quality is improving, and access is ubiquitous. The bottleneck has moved upstream. What’s missing is an execution substrate that can coordinate stochastic, goal-seeking agents at scale.

That substrate is the agent control plane: the orchestration, memory, auth, and tooling layers that allow autonomous agents to act reliably across real-world systems. Today, no one owns it. There’s no standard runtime, no dominant memory protocol, no consensus on agent identity or benchmarking.

This is the power vacuum. Whoever wins the control plane will define how agents are built, deployed, and governed—just as Kubernetes defined modern app orchestration. The biggest AI infra companies of this cycle won’t compete on inference. They’ll own the scaffolding that makes inference useful.

This “assembly line” approach to data labeling treated data as a mass-produced commodity — harvested at low cost and with minimal context, much like a raw industrial material.

That era is decisively over. However, building the huge business Scale did enabled the company to win this new era.

Data as a Bottleneck: A New Recruitment Challenge

Modern frontier models, particularly LLMs and multi-modal systems, have far surpassed the ability of basic crowd-sourced labels to train them. Instead, models now require high-context, domain-specific, and sometimes real-time feedback to stay competitive.

Instead of labeling dogs and cats, these systems must now learn:

• How to interpret legal briefs

• How to reason through medical diagnoses

• How to code correctly across dozens of frameworks

• How to explain financial statements

This means recruiting specialists — coders, financiers, lawyers, doctors — who can supply precise, context-rich annotations. Managing these highly paid, scarce human resources is no longer a simple “click to hire” task. Instead, coordinating them has become one of the hardest problems in modern machine learning operations.

👉 One estimate from a major AI vendor suggests that expert annotators cost 20–40× more per hour than generic crowd workers.

Data as a Strategic Moat

As model architectures increasingly converge — everyone has a transformer, everyone scales to billions of parameters — the competitive edge has moved squarely to data pipelines. Data, alongside compute, has become the defining moat.

A remarkable illustration of this comes from Anthropic. In 2025, Anthropic physically destroyed millions of print books to scan them into digital training sets, then won a lawsuit defending that mass digitization as legal under fair-use doctrines (Ars Technica). This bold, even destructive, strategy demonstrates how desperate top labs are to build defensible data advantages.

Today, the largest AI companies are spending $1–2 billion per year on human-in-the-loop reinforcement learning and other data-collection pipelines. That spend is only growing, with some forecasts suggesting data-labeling budgets could double by 2027, reaching over $10B annually across the top 10 labs. Today this number is about $5B.

Vendor Fragmentation and Neutrality

Previously, major players like OpenAI, Anthropic, Google, ByteDance, Apple, and xAI leaned on one or two major data-labeling partners — often Scale AI — to run their pipelines. But after Meta’s enormous $15B investment for a 49% stake in Scale AI, the landscape fractured. Rivals, worried about vendor lock-in and neutrality, have diversified their labeling partners:

• Micro1 (engineering and healthcare-focused labeling)

• Turing (specializing in complex RL workflows)

• Surge (high-quality human reinforcement learning feedback)

• Datacurve (niche coding domain experts)

These vendors have carved out specialties, providing higher-skill annotation for coding tasks, legal reasoning, and financial audits. Instead of treating data labeling as a single interchangeable commodity, labs are now orchestrating entire supply chains of human expertise across multiple providers to minimize risk and optimize for data quality.

👉 Think of it like diversifying your suppliers in a geopolitical crisis — with the supply being domain experts that capture nuanced reasoning of domain experts.

Research Speed as the Differentiator

In today’s AI competition, the rate of experimentation and iteration has become the critical differentiator. Reinforcement learning workflows — guided by expert supervision through authentic, real-world scenarios — have become central to staying competitive. The labs that can design, test, and refine these loops fastest will dominate.

Companies like Turing have built advanced orchestration platforms, often called “RL gyms,” to manage these cycles at scale. These systems coordinate human labelers, evaluation frameworks, and live experiments, dramatically shortening iteration times. In practice, this can let a lab validate and adapt three times as many hypotheses in the same window, sharply improving their odds of discovering breakthroughs.

The lesson is clear: in startups, speed to achieving product–market fit determines winners in startups; in AI, it’s the speed of adapting data to a model’s needs that sets leaders apart. The entire research engine must move quickly and flexibly, turning high-quality expert feedback into tangible model gains faster than rivals. That agility is increasingly the deciding factor in who stays ahead.

Source: Artificial Analysis [link]

Outlook: Domain-Specific AI is built on Data Strategy

If the 2010s were defined by breakthroughs in model architecture, and the early 2020s by scaling billion-parameter transformers on vast GPU clusters, the late 2020s will be dominated by data strategy — especially for domain-specific AI companies. In these more specialized areas, where models must understand legal reasoning, medical decision-making, or advanced coding tasks, data is no longer just fuel — it is the moat.

• Proprietary, high-quality data will be the ultimate barrier to entry

• Domain experts will be the most constrained and valuable resource

• Research velocity — the ability to quickly iterate on high-signal, domain-relevant data — will define competitive advantage

The field has moved beyond simple tasks like labeling stop signs, now prioritizing richly detailed expert reasoning with deep contextual nuance. Managing and coordinating specialists, compressing reinforcement learning loops, and building truly defensible data pipelines will separate the leaders from the laggards.

For founders and investors, the lesson is clear: owning the right data — the rarest, highest-impact, domain-specific data — is what will define the next generation of winners in AI.

At the center of this transformation is bandwidth. Training LLMs and serving real-time inference workloads at scale requires unprecedented interconnect density and performance—between GPUs, switches, CPUs, and storage systems. Each NVIDIA H100 server can burn 700W+ per card and demands 3.2Tbps+ of bi-directional communication. That’s not just a compute challenge—it’s a connectivity one. How do you move data at the required speeds to feed AI models the information they need?

Credo Semiconductor has quietly emerged as one of the most essential—but underappreciated—pieces of this new AI infrastructure stack. Unlike companies building the GPUs or LLMs, Credo is building the lanes the AI traffic rides on.

Credo’s Role in the AI Data Center Stack

1. High-Speed SerDes and Retimers

At the heart of modern AI servers is the need to move enormous volumes of data between GPUs, CPUs, and memory with minimal latency and signal loss. Credo’s SerDes IP (Serializer/Deserializer intellectual property) and retimer chips enable this by supporting PCIe 5.0, 6.0, and emerging PCIe 7.0 standards, as well as CXL (Compute Express Link)—all essential for high-throughput, low-latency connections inside GPU servers. PCIe 6.0, for instance, runs at 64 GT/s per lane, meaning a 16-lane (x16) link can deliver up to 256 GB/s of bi-directional bandwidth. But signal integrity degrades over long traces and dense boards—especially at 112G and 224G line rates. That’s where Credo’s retimers come in: they clean and re-amplify high-speed signals, restoring data integrity between chips, boards, and system components.

This is foundational in GPU-rich architectures like NVIDIA NVLink, Broadcom Tomahawk switching fabrics, or AMD MI300-based systems. Without high-performance SerDes and retimers, these clusters don’t scale.

2. AECs as Optical Alternatives

While optical transceivers dominate at 500m+, Active Electrical Cables (AECs) are emerging as a cost-effective, power-efficient interconnect for short reach (<5m) links. Credo’s ZeroFlap™ AECs offer >100× reliability improvements over active optical cables (AOCs) and have quickly become the go-to solution for top-of-rack to server interconnect in AI clusters.

As hyperscalers shift toward co-packaged optics and tighter thermal envelopes, AECs offer critical advantages in density, cost, and thermals. Credo is now considered a category-defining vendor in this segment.

3. PILOT: Software as a Differentiator

Credo’s PILOT platform, launched in 2024, adds observability and diagnostics to AEC and DSP links—making its products more “software-defined.” This aligns with the broader trend in infrastructure where hardware alone isn’t sticky enough. PILOT gives hyperscalers deeper telemetry and reliability—essential for maintaining massive AI fabrics.

Financial Performance: AI Product-Led Acceleration

• Annualized Revenue: $680M, up 26% quarter-over-quarter and 180% year-over-year

• Gross margins: 67%

• Annualized net income: $148M

• Market Cap: $12.4B, up 148% year-over-year

The growth is almost entirely product-driven—particularly from AECs and SerDes retimers, which hyperscalers are deploying at massive scale. Licensing (historically ~15% of revenue) has become less relevant as products shipments dominate sales.

Strategic Positioning and Conclusion

Credo Semiconductor exemplifies the second-order beneficiaries of the AI boom—companies not training models or building LLMs, but infrastructure for them to scale. Their mix of differentiated silicon, system-level innovation, and increasing software control gives them a long runway in hyperscale AI infrastructure.

• Category Creator in AECs: Credo leads a fast-growing segment with IP and manufacturing advantages.

• Vertical Integration Without Capex: Uses a fabless model but maintains control over ASIC and packaging design, balancing gross margin and capital efficiency.

• Embedded in Hyperscaler Pipelines: With adoption across AWS, Microsoft, and now reportedly xAI, Credo is becoming core to 800G+ and 1.6T data center designs.

• Long-Term IP Flywheel: As PCIe 7.0 and CXL 3.0 adoption grows, Credo’s IP licensing business may see a second wind—especially among custom silicon teams at hyperscalers.

For VCs, Credo is the kind of company that would be nearly impossible to back at seed today—but serves as a reminder: the AI stack is broad, and the infrastructure tail is long. Tomorrow’s Credos will emerge from similarly narrow technical wedges—SerDes, interconnects, thermal systems, memory fabrics—and quietly compound into critical AI infrastructure leaders.

The Traditional Model: Location, Location, Location

Source: Data Center Map

The traditional data center industry in the U.S. was built around proximity to people and networks. Most facilities were clustered in population-dense regions like the Northeast Corridor—especially Northern Virginia, where Ashburn alone hosts over 250 data centers and handles over 25% of U.S. traffic. In fact, Ashburn is known as “Data Center Alley” These locations were optimized for latency-sensitive workloads like cloud, SaaS, and mobile apps, prioritizing fiber interconnects and enterprise adjacency over raw power or physical scale. Operators like Equinix and Digital Realty built premium facilities near end users to support high bandwidth requirements.

Source: S&P Global

The 2015 Shift: The Rise of the Energy-First Data Center with Cryptocurrency Mining

Around 2015, crypto mining exploded. For the first time, a new kind of data center emerged—built not for people, but for algorithms. Crypto miners didn’t care about being near a user — or location at all frankly. They cared about:

1. Cheap, abundant electricity

2. Cheap/remote and large land lots to host racks of Nvidia GPUs

3. Minimal human intervention

It didn’t matter whether a Bitcoin was mined in North Dakota, China or Iceland. The physical location was irrelevant as long as the power was cheap. Companies like CoreWeave mined Ethereum using consumer-grade Nvidia GPUs; Crusoe converted flared gas from oil fields into compute, sourcing cheap excess electricity; Lambda Labs sold GPU boxes to miners before pivoting to the cloud.

Source: Meet the Company Mining Bitcoin Using the Flare Gas from Oil Drilling - and Drawing Investment from Coinbase and the Winklevii [Link]

2022–Present: AI Workloads Look a Lot Like Crypto

By 2022, the data centers built by crypto miners became incredibly valuable. AI training and inference workloads—driven by massive foundation models—demanded the same things that crypto mining data centers focused on to maximize profits:

1. Huge amounts of cheap electricity

2. Physical real estate and space to scale horizontally

3. High-performance Nvidia GPUs (H100s, A100s) to mining and now AI

Unlike cloud or SaaS workloads, AI compute is batchy, power-intensive, and capital-heavy. The new bottlenecks were no longer fiber routes or real estate density—they were power, GPUs, and cash. As AI training spend exploded (OpenAI’s cluster spend now exceeds $2B/year), the mining-born companies were already optimized for this. They had huge capacity before traditional clouds could build these AI data centers.

Source: Texas Miner pivots from Crypto to AI [link]

From Flare Gas to Foundation Models

Crusoe’s pivot is emblematic. In April 2024, Crusoe sold off its Bitcoin mining business to NYDIG to go all-in on AI infrastructure, building a 1.2 GW data center campus in Texas. This isn’t just a rebrand. It’s a recognition that AI workloads are now more profitable than crypto and better aligned with long-term capital markets.

“Crusoe’s vertically integrated model was built for a world of compute scarcity.” – Crusoe Press Release, April 2024

Source: Bloomberg, Powering Crypto Mining through nat gas flaring

Conclusion

In this new era, the crypto miners (CoreWeave, Crusoe, Core Scientific, and Applied Digital) didn’t just pivot—they already won. They owned the land, the energy contracts, the GPUs, and the capital mindset to build at scale. They had (inadvertently) optimized their data center footprint for exactly what AI workloads needed:

1. Cheap electricity

2. Huge physical footprints

3. NVIDIA hardware

This data center archetype has a fundamentally different archetype than the data centers for internet/mobile/cloud: near population centers, with higher energy cost and smaller physical footprints.

What MCP Actually Does

Model Context Protocol (MCP) is an emerging specification that solves this. It ensures that models operate with the right identity, the right access, and with full auditability, even in complex multi-tool workflows. MCP acts like an operating system for AI data access. It gives AI models the rules of engagement when interacting with enterprise systems. Specifically, MCP:

• Tracks Who’s Asking: Every AI request is tied to a real user or service account, so you always know who’s behind the query.

• Filters What’s Visible: The model only sees data the user is authorized to see—no more, no less.

• Orchestrates Complex Actions: It handles sequences of steps—like fetching files, calling APIs, and updating systems—while maintaining context across tools.

• Applies Enterprise Rules: Policies like “Finance data can’t leave the US” or “Only managers can view salaries” are enforced by design.

• Logs Everything: Every data access and decision made by the model is recorded, making it easy to audit or debug.

Source: Descope’s piece on MCP [link]

The Data Permissioning Challenge

In traditional SaaS:

• Users authenticate into one app (e.g., Workday).

• Permissions and data access are tightly scoped.

In modern AI:

• A user asks a model like OpenAI to answer a question.

• The model queries multiple systems (Workday, Salesforce, Jira, etc).

• But user permissions are unclear, and access control is often bypassed.

Why AI needs a data access layer

• In AI-forward orgs, API-based data requests exceed human ones by over 10x (47% of surveyed orgs).

• API data retrieval is expected to double in the next 12 months.

• Without identity-aware access control for these AI-triggered API calls, data leaks become inevitable.

Example of MCP-based AI agent request

Imagine you tell an AI voice assistant (think a customer service rep) for Delta Airlines that you want to change your flight:

"I want to change my flight to 9 a.m. tomorrow"

That one request then needs to touch the right tools to actually execute the workflow and change your flight. The first step is mapping out the tool requests:

• Speech-to-text (Deepgram)

• Intent detection (OpenAI)

• Authentication (Auth0, Okta)

• Flight pricing APIs (Amadeus, Sabre)

• Payment (FIS)

• Notifications (Twilio)

MCP sits between all of them, ensuring permissions, audit logs, and correct execution. Without MCP, any one step could access the wrong data or act outside user intent.

Adoption Trends of MCP

Model Context Protocol was introduced by Anthropic in November 2024 and quickly gained traction, especially after OpenAI integrated it into its Agents SDK in March 2025. Other industry players adopting MCP include:

• Google DeepMind and Microsoft have since joined the ecosystem, with Google confirming MCP support for its Gemini models and Microsoft integrating MCP into Azure Copilot.

• Developer-centric platforms like Replit and Sourcegraph have adopted MCP to enhance real-time codebase access and streamline software development.

• Block (formerly Square) have leveraged MCP to securely link internal AI systems to proprietary data.

• Cursor has also added MCP to connect internal tools and data to its application.

As of mid-2025, over 1,000 community-built connectors are available, making MCP an increasingly central standard for enterprise AI integrations. Industry analysts estimate that over 20% of enterprise AI deployments by the end of 2025 will utilize MCP or similar frameworks, highlighting its rapid adoption and strategic importance in secure AI workflows.

Why MCP Matters

MCP turns AI from a rogue tool into a trusted enterprise agent:

• ✅ Security: Stops over-permissive data access

• ✅ Compliance: Ties every request to identity and policy

• ✅ Scalability: Works across all apps, APIs, and users

In short, MCP is the missing glue between LLMs and real enterprise workflows.

TL;DR: Model Context Protocol (MCP)

As the AI stack matures, Model Context Protocol is becoming foundational—much like OAuth was for web apps.

Data and AI infrastructure are undergoing a tug of war, balancing the convenience of integrated platforms with the flexibility of modular, AI-optimized systems.. Data pipelines have moved beyond analytics and are now the foundation of any AI system. Over the past 10 years, we saw the rise of data platforms like Snowflake, Databricks and BigQuery, as well as Modern Data Stack tools around platforms like dbt, Fivetran, Looker and many others. However, the cost of Snowflake and Databricks remain high—particularly as AI workloads increase demand on compute.[link]:

“Databricks is fabulous if you can afford it… everyone loves Snowflake, but you can run a $60k query pretty easily by accident.” – CIO at F2000

From 2010 to 2022, Data and ML budgets massively grew unchecked, fueled by the Big Data movement and the widespread adoption of scale-out architectures. Now, however, CFOs are demanding greater ROI on AI and data infrastructure spending, leading to a renewed emphasis on cost efficiency and infrastructure optimization. This shift has opened the door for startups developing novel, cost-efficient architectures, making it a compelling investment theme. While large platforms provide convenience, they are often not cost-effective, and specialist vendors are emerging to drive major efficiency gains. Cheaper, workload-specific solutions are now defining the next wave of successful competitors. From foundational data management (compute and storage) to specialized workloads (query engines and geospatial analytics), new vendors are taking advantage of this trend.

As companies reassess their data and AI spending, a key architectural shift is making cost efficiency more attainable: the decoupling of storage and compute. Traditionally, data warehouses bundled these functions together, forcing businesses to scale compute alongside storage, even when unnecessary. The rise of Open Table Formats (OTFs) like Apache Iceberg [“What is Apache Iceberg?”], Delta Lake, and and even OpenTelemetry [“What is OpenTelemetry?”] are contributing to this unbundling by promoting open interfaces and ecosystem interoperability. These new data formats are breaking down platforms, allowing companies to store data independently and choose the best compute engine for each workload. This shift not only reduces costs but also creates a more flexible and competitive ecosystem where specialized query engines and workload-specific databases can thrive.

With storage and compute now decoupled, companies can optimize compute resources for specific workloads instead of relying on one-size-fits-all data platforms. New query orchestration and execution platforms are emerging to ensure workloads run on the most efficient and cost-effective compute layer with scale-up architectures. At the same time, specialized databases built for log data, time-series analytics, and geospatial workloads are providing lower-cost, high-performance alternatives to traditional warehouses.

As cloud data and compute costs continue to rise, this shift toward modular, workload-specific, cost-efficient architectures is defining the next generation of data infrastructure:

1. Data Transformation Efficiency – Cutting warehouse transformation costs

• Data transformation can be a major cost driver in cloud warehouses, especially when entire datasets are recomputed unnecessarily. SQLMesh and Tobiko Data reduce costs by 50% over dbt by using incremental processing, meaning only changed data is recomputed. This approach reduces redundant queries, lowers compute costs, and speeds up processing without affecting data accuracy.

2. Specialized Data Types and Workloads:

   • Observability Data – Manage and query high-scale log and time-series data

     

     • Storing and analyzing log and time-series data is often many $Ms per year because warehouses like Snowflake and BigQuery charge high fees for storage and compute. Hydrolix cuts costs by >75% with its compressed storage architecture and only using compute when needed, instead of keeping it running constantly. This allows companies to retain more data and run fast queries at a fraction of the cost.

Geospatial Data – Manage and query large-scale location data

• Warehouses are not built for geospatial workloads, making queries slow and expensive. Wherobots and its open-source Apache Sedona provides a cloud-native spatial database designed to handle large-scale geospatial queries efficiently, reducing compute costs while improving performance. By optimizing query execution and resource usage, Wherobots enables faster, cost-effective location analysis.

3. Queries: Management, Orchestration, and Execution Optimization

   

Credit: SeattleDataGuy’s What is Apache Iceberg piece

• Query Management & Compute Orchestration – Optimizing query execution across engines

  • Running queries across multiple engines is expensive, and most companies either manually manage it or stay locked into a single vendor like Snowflake. New startups are emerging to automate query routing, selecting the most cost-efficient engine for each workload, reducing unnecessary compute usage. Instead of overpaying for a single platform, companies can distribute workloads more efficiently across different processing engines.

• Query Execution Optimization – Moving to scale-up SQL

  • Running SQL queries in the cloud is expensive, especially when workloads rely on scale-out architectures like Databricks, Snowflake and BigQuery, which distribute queries across multiple compute nodes, driving up costs.

  • However, hardware advancements and the rise of Python-first workflows are enabling a shift back to scale-up computing, where local machines handle more processing before relying on the cloud. Today, even consumer-grade hardware like MacBook Pros can run 70-billion parameter models (equivalent to GPT-3.5) locally, demonstrating how much compute power is now available outside of large cloud clusters. This shift allows developers to prototype, train, and analyze data on local infrastructure, reducing cloud costs and unnecessary complexity.

  • Motherduck optimizes costs by running queries locally first (scale-up) before leveraging cloud compute only when necessary.

  • Similarly, DataFusion (7K+ stars), an in-memory query engine built in Rust using Apache Arrow’s columnar format (detailed below), offers fast, cost-effective SQL execution without the overhead of cloud warehouses, making it well-suited for scale-up processing.

  • Apache Arrow (15K+ stars) has built a universal connectivity layer for modern data warehouses to eliminate inefficiencies from custom connectors, costly middleware, and manual data movement. Some engines optimize query orchestration by routing workloads (where to run) to the most cost-effective engine, while others enhance execution efficiency (how queries execute) —together, they reduce compute and improve performance.

Intro:

After 18 months since ChatGPT’s release and OpenAI’s APIs reaching enterprise use cases, we now have a good sense of what LLMs are best suited for: searching, retrieving and parsing enormous data sets to automate repetitive tasks, leveraging domain-specific knowledge. To find where LLMs will have enterprise use cases, look for the largest data sets (“data gravity”): customer support, domain knowledge, logs and events, customer and operational data, sales and marketing data, and IT infrastructure data are some of the largest.

In response to the rise of foundation models and the vast training data sets powering them, enterprises and technology vendors are increasingly investing in their own observability and telemetry data to build defensibility. By collecting unique, deeply integrated product usage data, they create proprietary data assets that are difficult to replicate by foundation models. As data becomes more commoditized, this truly owned data is becoming a vital competitive advantage. With the rise of data sources and cloud applications, security as a field has become more like data engineering.

Logs and Events are typically the largest volume data set in the modern enterprise (often petabytes per day), from sources such as system and application logs, security events and network traffic. These logs are mission-critical for monitoring, diagnostics and compliance. These data sets get enormous over time and create major data engineering problems. Logs and events are also real-time sensitive. Overage costs can also be a nightmare for executives, who have limited predictability on data volumes and thus costs. It’s not uncommon for F500 enterprises to spend tens to hundreds of millions on observability data: OpenAI is paying Datadog in excess of $100M, with Coinbase paying $65M/yr in 2023 and Capital One paying >$50M. Thus, logs and events stand out as a prime use case to insert an LLM due to the sheer volume and frequency of data generation across all systems and applications.

The observability market is fragmented, with leading vendors like Datadog, Splunk, and Dynatrace each holding less than 20% of the market share and together boasting over $10B of revenue. Further, the number of Cribl customers sending data to multiple SIEM products increased 45% year-over-year as different data sources require different destinations and lock-in has reduced with new storage formats [Cribl Report]. Microsoft Sentinel has crossed $1B of ARR in 3 years since its launch. The total observability spend is in excess of $30B and is reasonably fragmented. A Gartner survey found the average F2000 has between 7-10 observability tools, each with its own query language and data model. The number of data sources is growing 32% year-over-year and over a third of Cribl customers are consuming data from ten or more sources. Modern distributed systems generate petabytes of telemetry data daily, in various heterogeneous formats such as logs, metrics, and traces – all across different tools.

While this works for storing the data, the intelligence layer is fragmented across data silos. The real challenge isn’t just in collecting or storing this data but in making sense of it quickly enough to drive real business value. Data dimensionality, the rise of open telemetry and open table formats in storage are key trends reshaping the market. Complex data challenges become far more addressable with AI and LLMs, which can read in heterogeneous data formats.

Basics on Data Schema for Security Observability:

Observability is the thorough process of collecting and analyzing data to understand system performance, made up for three pillars. Each of these three pillars has its own data types and format, historically requiring its own querying engine and specialized storage.

1. Logs

• Nature: Semi-structured data capturing events within a system, extremely voluminous as detailed information on user actions, system errors, access logs

• Key attributes:

  • Structured Data: Timestamps, severity level, service and instance information, tracing information, basic user and request info, basic error details, metadata.

  • Unstructured Data: Detailed event descriptions, detailed user and request info, detailed error messages.

• Use cases: Debugging, auditing, tracking errors, server and applications capturing each request and response

2. Metrics

• Nature: Structured, numerical data representing quantitative system state over time, less voluminous than logs as metrics are numerical data points at regular intervals representing system states

• Use cases: Monitoring system performance and tracking resource usage like GPU/CPU usage, memory consumption, request rates, error rates

3. Traces

• Nature: Semi-structured, moderate volume (much less than logs) as captures detailed information about the flow of requests through a system

• Structured attributes: Trace ID, span ID, parent span ID, service name, operation name, timestamps, duration, status code, resource info

• Unstructured attributes: annotations like custom messages or status updates, logs, tags/labs like error descriptions to provide additional context, metadata about the operation like user environment

• Use cases: Capturing the flow of a user request through multiple microservices, including time spent in each service; to be used for performance optimization, identifying bottlenecks and understanding end-to-end request flows

Data Dimensionality:

In the realm of data processing and storage, particularly for log data, the dimensionality and complexity of the data significantly influence how it is managed. Hydrolix, for example, specializes in handling large, multi-dimensional transaction logs such as CDN logs, which capture entire user sessions, including detailed activities on platforms like Disney+. These logs are not only vast in size but also rich in context, making them critical for long-term storage and analysis without any data loss. Hydrolix’s approach is built to accommodate the high-dimensional nature of these logs, ensuring that every piece of data is retained and accessible for years, which is essential for compliance and in-depth analytics.

On the other hand, traditional application logs, which are smaller and less complex, are often generated by microservices and containerized environments. These logs are typically high in frequency but lower in individual value, leading to the development of solutions like Cribl, which intelligently filters and decimates less valuable logs, metrics, and traces. Cribl’s method involves identifying and discarding low-value logs while forwarding the more relevant data to platforms like Splunk for further analysis. This approach contrasts with Hydrolix’s, where the goal is to preserve the integrity of all data due to the higher dimensionality and value of transaction logs. Together, these strategies highlight the importance of tailoring data processing and storage techniques to the specific nature and dimensionality of the data being managed.

Where are LLMs useful in observability?

Thus, given logs have the highest data volumes and the most unstructured text-heavy data attributes, LLMs are the most natural fit here. LLMs should be highly effective at parsing and analyzing the text-heavy nature of logs with NLP. LLMs can do the following with logs:

• Extract Info: Identify and extract key entities, error messages, and patterns from logs

• Anomaly Detection: Detect unusual patterns or anomalies that indicate system issues.

• Summarization: Generate summaries of log data to highlight critical events and trends.

• Log Categorization: Classify and tag log entries to streamline searching and analysis.

Metrics have limited usefulness with LLMs as they are typically structured and numerical. Expect incumbents to integrate the few useful advances from LLMs like natural language summaries of metrics trends or anomalies, or predictive analysis based on trends in data.

Traces have medium usefulness, with some excellent and defensible use cases given the structured identifiers mixed with semi-structured annotations at moderate data volume. For traces, LLMs can better perform:

• Root Cause Analysis: Identify patterns in trace data that may indicate performance bottlenecks or errors.

• Trace Analysis: Understand and summarize the flow of requests through various services.

• Correlation and Context: Provide contextual insights by correlating trace data with logs and metrics to offer a comprehensive view of system behavior.

• Predictive Maintenance: By analyzing logs and patterns, LLMs can predict system failures or performance issues before they occur.

Surprisingly upon digging in, many of the state of the art techniques for data engineering in security are highly rules-based and interpreted from the data schema of event data from customers, rather than being based on a trained AI model. The rules layer originally started with rules for cost savings, such as which data to route to cold storage and cheaper systems than Splunk, but opportunities also exist for processing with a focus on escalation. Cribl has built a $200M ARR business growing 70% and valued at $3.5B in applying these principles of data engineering (traditionally found in ETL for warehouses) to security data systems. This is their secret sauce, which is an alternative to custom configuring Kafka rules.

Security Data Engineering (Today):

This diagram outlines the data flow from initial sources like Network Data, Identity Data, Cloud APM Data, CDN Data, and Infrastructure Security Data through to data processing, streaming, and finally to storage in SIEM systems or Data Lakes. Data begins at these source points, where it is generated in huge volumes; as mentioned, the largest data sets in most enterprises which creates lots of challenges. Cribl’s report shares insights on the most popular data sources it sees, with most of its enterprise customers using more than 10 different data sources. Splunk is most popular across the board, while other tools like S3 are popular with fast growing companies and O365/Windows Event Logs are most popular for Enterprise. You’ll notice the flow looks a lot like traditional data eng around Snowflake/DB.

Historically, data transformation has been limited as the sheer costs are prohibitive. Thus, tools in the “Preprocess, Filter, and Enrich” category (e.g., Cribl, Splunk DSP; Observo, Tarsal and Databahn as the new startups) are relatively new, with new techniques to clean, filter, and prepare this data, ensuring it is optimized for further use. These preprocessing companies are the most opportune for startups, as they can play neutral across vendors, have a clear cost savings ROI and also have a clear quality ROI. There is surprisingly limited use of AI in transformations here.

Once processed, data often moves into “Data Streaming” platforms (e.g., Kafka, Pulsar, Flink) that handle real-time data flow, critical for applications requiring immediate insights. Finally, the data is stored in “Traditional SIEM” systems like Splunk and Elastic or in “Data Lakes” such as Snowflake and Databricks, where it can be analyzed and retained long-term. The map also highlights emerging players in “Next-Gen Data Lake/SIEM” like Hydrolix and Runreveal, which are designed to manage modern data demands with advanced analytics and storage capabilities. This structured flow ensures that data is efficiently managed from generation to storage. Many of these next-gen SIEMs have similar product marketing on cost savings, just through different solutions like storage formats.

Much like the data sources, we see Splunk (majority of Cribl customers) and S3 topping the list as most popular used destinations. However, we see a growing fragmentation away from Splunk’s historical dominance on the destinations side with CrowdStrike’s Falcon SIEM, Azure Logs (via Sentinel) and Google SecOps each growing >250% in data volume across the Cribl user base.­­­­­ There was a 73% increase in companies using multiple SIEM products this year. Destination systems are growing in fragmentation, with 90%+ of Cribl customers sending to 2+ destinations and 12% sending to 4+ destinations, with overall destinations growing 15% YoY.

The Splunk Architecture: Forward, Indexer and Search Head

Splunk’s architecture is designed around three core components: the forwarder, the indexer, and the search head, each playing a critical role in data collection, processing, and querying. The forwarder is responsible for collecting data from various sources, such as sensors, APIs, and firewall appliances, and sending it to the indexer in real time; this is what Cribl has attacked head on. However, the forwarder is agnostic to the data it receives, meaning all data sent to the indexer counts against a user’s data allowance, regardless of its relevance or value.

The indexer ingests the data and builds an index to facilitate efficient querying. This process, however, comes with significant cost and performance challenges, especially in cloud environments. Tools like Cribl offer a solution by pre-processing data before it reaches the indexer, removing unnecessary fields and reducing storage costs. Cribl’s ability to send event data to cold storage can save companies up to 97% of their storage costs, significantly reducing the financial burden of managing large volumes of log data. Additionally, Snowflake and Databricks provide a compelling alternative for data storage and querying.. Its cloud-native architecture scales automatically and can offer querying speeds up to 200 times faster than traditional SIEM systems like Splunk, making it a powerful tool for organizations needing to analyze large datasets quickly. These advantages allow companies to streamline their data management processes, reduce costs, and improve the speed and accuracy of their security operations. Thus, this data engineering flow around the SIEM is being unbundled

Current Challenges:

In Security Data Engineering, buyers have focused on two main problems. One of the primary challenges is the ballooning cost associated with storing the vast and ever-growing volumes of security data generated by today’s distributed systems. This data, ranging from unstructured logs to events and telemetry to complex traces, often reaches petabyte scale, leading to soaring and unpredictable storage expenses. These costs are exacerbated by the lack of a unified data model, which forces companies to rely on specialized storage solutions and tools. This situation has resulted in fragmented systems and data silos, further complicating data management and driving up operational costs. When it comes to managing the dimensionality of data and how that affects the growing volume of security data, Cribl and Hydrolix take different approaches. Cribl focuses on decimating low-value, high-frequency application logs, intelligently filtering out less important data to optimize storage and reduce costs. In contrast, Hydrolix handles high-dimensional transaction logs, ensuring that all data is preserved without any loss, which is crucial for long-term analysis and compliance.

The second challenge is the difficulty in distinguishing critical security signals from the vast amounts of data noise. Retrieval and ranking of relevant information remain major challenges in observability, despite advances in data search capabilities. The ability to efficiently identify what and where to look within massive data volumes continues to be a bottleneck. The Mean Time to Resolution for critical incidents (MTTR) still averages 4-5 hours per research from Gartner. Traditional systems often fail to prioritize and filter effectively, making it hard to detect significant threats. The core issue isn’t just collecting or storing this data but making sense of it quickly enough to provide real business value—a problem that has often been treated as a big data issue rather than an intelligence one.

Further, modern observability tools encounter a variety of technical challenges, including handling heterogeneous data formats, and grappling with the absence of a unified data model. Iceberg and open table formats are making the storage problem less severe. With these tools, there becomes less need to duplicate data across different systems as the querying engine can read from multiple sources. For example, Cribl’s Search product can search across Splunk, cold storage, time series DBs and other security tooling because of the rise of open table formats. There may be an opportunity for companies like Lakeway to support these search use cases on observability data. Additionally, the diversity of query languages across different tools—such as Lucene for Elasticsearch, PromQL for Prometheus, and various SQL-like languages for tracing—adds another layer of complexity. This diversity makes it difficult for teams to effectively diagnose and resolve issues, as they must navigate multiple systems to get a complete picture of system health.

These challenges open up significant opportunities for companies like Cribl, Tarsal, Databahn and Observo, which are developing solutions to preprocess, filter, and enrich data, making it easier to prune useless data and to identify the most critical security signals – this attacks the core of Splunk’s forwarder technology. LLMs are particularly promising in this space because they offer a unified approach to data analysis. LLMs are well-suited to parsing and interpreting the unstructured, text-heavy data found in logs, and they can also incorporate context from system documentation, code repositories, and historical incident reports. This ability to adapt quickly to new data patterns makes LLMs a powerful tool for tackling the ongoing problem of concept drift in dynamic systems. Further, LLMs can be used as a policy/cost hygiene layer in addition to a semantic understanding layer. Depending on requirements, enterprises will have much more flexibility in addressing these challenges.

Traditionally, using AI for these tasks was seen as too costly, but recent advancements in open-source models are changing this. A new approach using a Llama-based log and event transformation model, hosted within a private environment, offers a scalable and cost-efficient way to enhance logs by identifying patterns, categorizing data, and prioritizing critical events.

This approach not only enhances threat detection capabilities but also optimizes resource allocation in secure environments, helping to resolve many of the fragmentation issues currently plaguing the observability market. However, it’s important to recognize that while LLMs are powerful, they come with their own set of challenges, particularly around real-time processing, which can be hindered by current latency and cost limitations. A promising future direction involves combining LLMs with graph databases, which could enable better automated root cause analysis and potentially reducing the Mean Time to Resolution (MTTR) by a significant margin.

Emerging Technologies: The Rise of Open Observability

1. OpenTelemetry: As data in observability becomes far more open, frameworks like OpenTelemetry are transforming the industry by standardizing the collection and transmission of telemetry data across various platforms. OpenTelemetry, an open-source observability framework, enables the seamless integration of different observability tools, allowing for the collection of data such as traces, metrics, and logs in a consistent format. This openness significantly reduces the risk of vendor lock-in and increases fragmentation of destinations, as organizations can now easily switch between back-end systems without the need to replace their existing data collection infrastructure. Read our piece “What is OpenTelemetry” to learn more.

2. Open Table Formats: Traditionally, observability vendors often locked customers into their ecosystems through proprietary data formats and collection agents, making it difficult and costly to switch providers. However, with OpenTelemetry, organizations gain the flexibility to select the best tools for their specific needs, promoting a more competitive and interoperable market. This shift is further reinforced by the adoption of Cribl Search, which complements OpenTelemetry by offering search capabilities across various destinations like S3 and other security data lakes, making it easier to manage and analyze observability data in a more open and vendor-agnostic environment. Read out piece “What is Apache Iceberg” to learn more"

3. Open-Source Models: Yet to see much application of small, domain-based models for preprocessing and routing events but expect this as mentioned above.

Startup Opportunites:

1) AI-First Security Data Engineering or “Security ETL”

• Description: Platforms to enhance security data engineering by focusing on efficient data preprocessing, filtering, and enrichment. Cribl reduces vendor lock-in by allowing organizations to route and process observability data across multiple platforms, such as Splunk, Datadog, and Elastic, enabling them to avoid dependency on a single vendor and choose the best tools for their specific needs. This is a big market that is growing faster than traditional ETL like Fivetran, with Cribl at over $200M of ARR and growing 90%. Particularly Splunk’s forwarder is being unbundled.

• Cribl began as a cost savings story for Splunk (where 75% of its data are synced to as destinations). Cribl promised to reduce Splunk bills by 30-40%.

• Instead of routing all data to Splunk, Cribl emerged to route some to cold storage, hot storage, or other systems like a time-series DB. This can save 97% if cold storage versus Splunk. [Cribl Docs]

• The emergence of open table formats has enabled Cribl to get into the search business, as Cribl search offers querying across various destinations. Because you can now query across various destinations, there’s less need to store everything in Splunk.

• Their solutions aim to reduce storage costs and improve the relevance of security data retained for analysis. These tools are most commonly deployed on-premise (i.e. ⅚ of Cribl) due to cost efficiency.

2) AI-Based Tracing and Root-Cause Analysis, and Event Correlation

• Observability tools are evolving beyond traditional metrics, logs, and traces by integrating code-level insights, which addresses significant gaps in root cause analysis. This shift allows for a more comprehensive understanding of system behavior by connecting performance data with the underlying code.

• New AI models, including LLMs, are enhancing observability by analyzing code, metrics, and logs within context, which reduces dependency on domain expertise and improves troubleshooting efficiency.

• Description: LLMs have the potential to revolutionize tracing by efficiently analyzing and correlating vast amounts of trace data, identifying patterns and anomalies, and providing deeper contextual insights. This capability can help teams more accurately and quickly pinpoint the root causes of issues. Moreover, LLMs can enhance accessibility by enabling natural language interfaces, allowing engineers to query systems using plain language. For example, asking a question like ‘Show me all HTTP 500 errors in the payment service correlated with high CPU usage in the last hour’ could yield immediate and accurate results, simplifying the debugging process and reducing reliance on specialized knowledge of query languages.

• New tracing techniques can automatically identify and resolve issues in complex distributed systems. Building on the advancements of companies like BigPanda, Moogsoft and Epsagon (acquired by Cisco for $500M), which represent the latest generation of event correlation and advanced tracing solutions, new startups are now exploring opportunities to further enhance the management and troubleshooting of large-scale, event-driven architectures.

3) Affordable and/or Open-Source Telemetry

• OpenTelemetry’s impact is extending beyond reducing vendor lock-in. The observability landscape is seeing increased fragmentation as companies utilize multiple tools simultaneously, such as Datadog, Prometheus, and Grafana, which are integrated through open standards like OpenTelemetry.

• Description: The high costs of traditional observability and security logging tools create significant opportunities for open-source solutions like Signoz. By providing an affordable, open-source alternative for observability, Signoz can replicate the success that Grafana had with attaching itself to Prometheus (Grafana recently raised at $6B on $270M of ARR). Notably, at the first Prometheus conference just five months after Grafana’s release, 30% of attendees were already using Grafana. Signoz is attaching itself to OpenTelemetry, which is the 2nd most popular CNCF open-source repo after Kubernetes.

4) Next-Gen Orchestration:

• Company: Maestro does not have a company yet

• Description: Maestro (Github) by Netflix is developing next-generation orchestration tools that streamline the management of complex, distributed applications, enabling more efficient and resilient operations. Experts like Bharat at Airbnb have mentioned Airbnb moving off Airflow (of note, Airflow was developed at Airbnb and led to Astronomer) towards Maestro and the market in need of new solution for orchestration.

5) Next-Gen Storage Formats

• Description: Nimble (Github) by Facebook is a new storage technology for columnar data (as an alternative to Iceberg, Hudi and Delta Lake). They claim significant cost savings particularly for ML and analytical workloads on event data. There is perhaps a gap in the market now with Iceberg / Tabular and Delta Lake owned by Databricks for something that is independent. This alone may not be big enough to build a platform company, but is a differentiated attribute that addresses the cost problem. Differentiated storage formats can be a big differentiator for innovation further up the stack, such as with Hydrolix and its cost efficiencies. This may be a differentiated entry point, overcoming the data gravity of incumbents.