No architecture is cheapest everywhere: cost per million tokens against tokens per second per user, with measured crossover points between architectures
In February, AMD’s MI355X served Qwen3.5 at 192 tokens per second per GPU. In May, the same GPU served the same model at 3,660. Nothing shipped from a fab in between; three pull requests landed in an open-source inference engine, and the chip got nineteen times faster.
Around the same time, SemiAnalysis measured a B200 serving DeepSeek R1 at 941 tokens per second per GPU in a standard 8-GPU server, and 4,130 when the identical die sat in a 72-GPU NVLink rack. Same silicon, same model, 4.4x apart.
What kind of product behaves like that? Not a chip. A chip’s performance does not quadruple with its packaging or multiply nineteen-fold with a software release. A system’s does. That is the argument of this piece: the accelerator cannot be evaluated apart from the system around it, because the system, not the die, sets cost per token. Most accelerator pitches I see still lead with a FLOPS bar chart. The measurements above are what that chart cannot see, and the end of this piece is a picture of the market they are quietly building.
So here is the claim in its strongest form. Training was one optimization problem, homogeneous enough that the fastest chip in the tightest cluster won nearly every time, and it built homogeneous fleets. Inference is many problems wearing one name. Models vary a hundredfold in size, contexts vary a thousandfold in length, some buyers need a token every seven milliseconds and some need a million tokens for a nickel, and prefill and decode stress opposite parts of the machine inside a single request.
Against that variety, each accelerator traces out a performance surface over the space of workloads, and the surfaces cross. There is no fastest chip, only chips that are fastest somewhere. Heterogeneous workloads make heterogeneous fleets, which is why Anthropic already serves Claude on TPUs, Trainium, and NVIDIA GPUs at once, and why the interesting question is no longer whether inference goes multi-silicon but who besides the hyperscalers gets to benefit.
Same silicon, different system: measured throughput multiples on identical or equal-FLOPS silicon when one part of the system changes, from 1.9x for bandwidth to 19x for software
By the numbers: the ridge point band, the H100 to H200 gain with zero extra FLOPS, same-chip fabric gains, software speedups on fixed silicon, the memory wall ratio, and the ASIC share of 2026 AI servers
How a Token Gets Made
To see why the FLOPS model fails, follow one request through the machine. It has two phases. First the model reads your whole prompt in a single parallel pass, called prefill. This is the compute-heavy part, and it is the phase GPU marketing implicitly describes. As it reads, the model writes itself notes about every token, called the KV cache, so it never has to reread the raw text.
Then comes decode, the phase that produces every visible token, one at a time. To emit each one, the GPU must stream every active model weight and the session’s entire cache out of memory, run a small amount of arithmetic on the way through, and repeat. At small batch sizes, decode performs about one or two floating-point operations per byte it loads, against the two to three hundred per byte the chip was built for. Until batching fills them, the math units spend most of the token-generating phase waiting on bytes.
That imbalance is not an accident of one design; it is decades of hardware history. Peak compute grew roughly 60,000x over two decades while memory bandwidth grew 100x, and the gap widens every generation: from A100 to Rubin, low-precision compute rose at least 56x against 11x for bandwidth. Training escapes the imbalance by batching enormously. Inference is where it gets billed.
The memory wall: compute throughput has outrun memory bandwidth every generation from A100 to Rubin, while the ridge point that decides whether decode is compute- or bandwidth-bound has barely moved
Hold that picture, a token is a memory read with some math attached, and the rest of the piece is a tour of the places bytes get stuck: the accelerator’s own memory, its bandwidth, the storage behind it, the wires between chips, and the software that orchestrates all of it. At every stop the ranking of chips can flip, and that is not a figure of speech. The reversals below are all measured.
Fit Beats FLOPS
The first constraint is the dumbest: does the model fit? Llama 405B at full precision needs about 1,053GB with working overhead. A node of eight H100s holds 640GB, so the model must be quantized or split across two servers, with every decode step paying the network toll between them. Eight MI300Xs hold 1,536GB. The model simply fits, with 480GB to spare.
The spare room matters more than it looks, because leftover memory holds the KV caches of concurrent users, and concurrency is what fills idle math units. A chip that fits more sessions runs bigger batches, and bigger batches convert wasted FLOPS into billable ones. When dstack benchmarked 405B in late 2024, with AMD’s software still visibly immature, the MI300X node produced roughly 1,970 tokens per second against 643 for the H100 node, which had no memory left to batch with. Three to one, for the chip with the weaker software and the lower street price. Capacity is compute you get to use.
Memory budget: Llama 3.1 405B at FP16 needs about 1,053GB with overhead; a node of 8 H100s holds 640GB and must quantize or split, while 8 MI300Xs or 8 H200s fit it whole
The H200 Experiment
If decode really is a memory read, a clean prediction follows: add bandwidth to a chip without adding compute, and tokens per second should rise anyway. NVIDIA ran that experiment and sold it as a product. The H200 is the H100’s exact die with more and faster memory attached, not one additional FLOP, and it delivers up to 1.9x the throughput on Llama 70B. Time to first token barely moved, because prefill is the compute-bound phase and prefill got nothing.
Rank today’s accelerators by measured decode ceiling and the order follows the bandwidth column of the spec sheet, not the FLOPS column. Push the logic to its limit and you get the strange machines at the market’s edge: Cerebras serves Llama 405B at 969 tokens per second per user from a wafer holding just 44GB, because that 44GB sits behind 21PB/s. And you get the strangest data point of all: in December 2025, NVIDIA licensed Groq’s SRAM-based architecture in a deal reported near $20B, and it now markets the result, the Groq 3 LPX rack, as a decode companion to Vera Rubin. The company with the most FLOPS in the world buying a bandwidth machine tells you which resource decode actually consumes.
Tokens per second follows bandwidth, not FLOPS: the H200 has 1.0x the H100’s compute and 1.4x its bandwidth and delivers up to 1.9x the throughput; across five accelerators the decode ceiling tracks bandwidth
Agents Changed the Bill
The number I keep coming back to is 126,000. That is the median prefix, in tokens, that a real coding agent reads per LLM call, measured across 4,265 sessions in TraceLab’s study, to produce about 252 tokens of output. The 90th percentile prefix is 467,000 tokens, roughly 149GB of KV cache on a Llama-class model. More than any accelerator sold, for one user’s session, and every constraint so far, fit and bandwidth alike, scales with exactly this variable.
That shape is money. At a typical $3-per-million input rate, rereading a 126K prefix costs about 38 cents per call, hundreds of times per session; served from cache at the standard 90% discount, about 4 cents. So the ranking of chips became a function of context: at 4K tokens the contest is bandwidth per dollar, at 128K it is who fits more users in memory, and at agentic scale the cache does not fit in memory at all. At that point the accelerator’s most important competitor is not another accelerator.
KV cache versus context length, log scale: five current model architectures diverge nearly 10x in cache per token, and by one million tokens even the most efficient exceeds a B200’s capacity
Storage Becomes Compute
It is the storage tier. Once inference reuses state, storage and memory become substitutes for compute: a prefix the system has seen before can be recomputed on a $30,000 GPU or read back from flash, and reading wins by an order of magnitude. The 90% cached-token discounts at OpenAI, Anthropic, and Google are that substitution, printed on a price sheet. DeepSeek, which has kept its cache on SSDs since 2024, discounts 97%.
The engineering results say the discounts are economics, not marketing. Lablup and VAST cut warm-turn time to first token on 140K-token agent contexts from 22.0 seconds to 6.6 by serving the cache from external flash; LMCache took Qwen3-235B from 3.98 seconds to 0.29 with a host-DRAM tier. Identical GPUs before and after. TrendForce attributes the record $18.5B enterprise-SSD quarter in early 2026 directly to AI agents, and NVIDIA now publishes a four-level hierarchy telling operators where to park state, from HBM down to networked flash. When the GPU vendor writes a storage architecture, storage has joined the inference system.
The Fabric Tax
Frontier models stopped fitting in one server, so every decode step now includes a conversation between chips, and the price of that conversation depends on which side of a cliff you stand. Inside a rack-scale domain, a GB200 GPU reaches any of 71 peers at 900GB/s. The moment traffic leaves an 8-GPU box for ordinary 400G networking, the figure is about 50GB/s. Eighteen times apart, and invisible on any spec sheet, because it is a property of the wiring, not the chip.
The 4.4x from the opening is this cliff, measured. Mixture-of-experts models like DeepSeek R1 scatter hundreds of experts across GPUs and shuttle every token among them mid-inference; the rack’s fabric makes that cheap, the box’s fabric makes it ruinous, and in SemiAnalysis’s like-for-like disaggregated comparison, built to isolate exactly this variable, the same B200 die lands at 4,130 versus 941 tokens per second per GPU. The corollary explains NVIDIA’s own behavior better than any keynote: a company that once sold cards now sells 72-GPU domains, and its networking business booked $14.8B last quarter, up 199%. The wires became part of the accelerator.
Same chip, different fabric: B200 in an 8-GPU HGX node versus GB200 NVL72 on DeepSeek R1 FP4; the tokens-per-GPU gap widens from 1.7x to 4.4x as interactivity rises
The Software Month
Which leaves the 19x, and its resolution is the most unsettling of all, because the component that moved was invisible. The three SGLang pull requests behind the MI355X’s jump were a fused expert kernel, a rewritten normalization path, and a fused top-k. That is the entire changelog between a chip that looked uncompetitive in February and one that looked excellent in May.
This is not an AMD story. The B200 got five times faster on gpt-oss-120b in two months as new quantization and speculative decoding landed, taking NVIDIA’s own cost per million tokens from $0.11 to $0.02. Meanwhile the H200 running TensorRT-LLM did not move for four months; that pairing had already reached its ceiling. So software gains are enormous, uneven across vendors, and unpredictable in timing, and two conclusions follow. Software maturity deserves a column on the spec sheet, where it is the most defensible thing CUDA has. And every benchmark is a snapshot of hardware plus that month’s software. A ranking from October is not a ranking in May.
Same silicon, different month: measured tokens per second per GPU on fixed hardware before and after software releases, log scale
There Is No Fastest Chip
Assemble the whole machine. Fit, bandwidth, state, storage, fabric, and software each move throughput by integer multiples, each binds a different workload, and none of them appears in a FLOPS comparison. FLOPS still matter; they are one constraint among six. The precise claim is that peak FLOPS lose their explanatory power the moment any other constraint binds, and in inference another constraint almost always binds. A workload is a point in a space; each accelerator is a surface over that space; the surfaces cross. In the current benchmark record alone: MI325X undercuts H200 on Llama 70B cost at every interactivity level, MI355X undercuts B200 on gpt-oss below 225 tokens per second per user and loses above it, and the rack-versus-box crossover moved between rounds because the software on each side changed at different speeds.
The accelerator map: current inference silicon compared on dense FP8 compute, HBM capacity, bandwidth, scale-up domain, software maturity and best-fit workload region
Two facts, then, and everything from here follows from them. Performance is a property of the system, so no single architecture should be expected to dominate every workload. And inference is becoming stateful, so the workloads themselves are diverging: chat, agents, batch, and latency traffic now stress the machine in different places. Put the two together and the familiar question dissolves. Asking which inference chip is best is like asking which aircraft is best. A 787 and an A321 are not competitors; they are answers to different routes, and no serious airline runs one airframe. Carriers match aircraft to routes, obsess over load factor and fuel, and treat the fleet mix itself as the strategy. Load factor even has an exact analog here: batch utilization, the share of a chip’s paid-for compute that passengers actually fill. Inference compute is becoming an airline business, and almost everyone is still shopping for the one best plane.What the Fleet Looks Like in 2029
Walk into a large inference facility in 2029, as I expect it to be built, and the GPU warehouse is gone. One hall runs rack-scale domains, NVLink or its Ethernet rivals, serving the frontier mixture-of-experts models that need seventy chips talking at 900GB/s. The next hall is high-memory nodes packed with long-context agent sessions, chosen because fit beats FLOPS for that traffic. A third belongs to co-designed ASICs earning their keep on the narrow, stable set of workloads they were optimized around. Near the front, where the latency-critical traffic lands, a few aisles of SRAM machines produce tokens faster than a person can read them, the corner Cerebras’s 969 tokens per second per user already stakes out. And underneath every hall, the quietest and maybe most valuable layer in the building: a flash estate holding the accumulated context of millions of agents, exabytes of KV cache that get read back at 90 to 97% below the price of recomputing them.
Every element of that building is shipping or announced today. The mixed halls are Anthropic’s fleet: up to a million TPUs, over a million Trainium chips, NVIDIA capacity through the clouds, one model family across all of it. The commitments are OpenAI’s: ten announced gigawatts with NVIDIA, six with AMD, ten of Broadcom custom silicon, roughly two of Trainium, multi-year announced figures rather than deployed capacity, but a fleet plan by any name. The ASIC hall is Microsoft’s Maia 200 serving GPT-5.2 and Copilot. The flash estate announced itself in TrendForce’s numbers as a record $18.5B enterprise-SSD quarter attributed to AI agents. Even NVIDIA is building the mixed hall internally: by its own product materials, the Vera Rubin platform spans Rubin GPUs, the Groq-derived LPX decode rack, BlueField in front of storage, a Vera CPU underneath. ASIC-based systems are already 27.8% of 2026 AI server shipments and growing faster than merchant GPUs. The 2029 building is just 2026 with the ratios moved.
Now price the building. The airframers each keep the routes they are shaped for, and NVIDIA’s route is the biggest: the broad, changing middle of the workload space, where a buyer cannot predict its own mix and CUDA’s maturity is worth a real premium. Data center revenue of $75.2B last quarter, up 92%, is not a franchise in decline. But the margin structure of the building favors a different class of company, the ones every hall depends on no matter whose logo is on the racks. The serving engines that let one model run across five silicon families. The KV-cache storage tier that turns recompute into a read. The fabrics, NVLink on one side and the UALink and Ethernet coalition on the other, that set how large a coherent domain can get. Airlines fought each other to thin margins for a century while the airports, the leasing companies, and the reservation systems compounded. The picks and shovels of a fragmenting market are the layers that make the fragments composable.
Who gets to operate a building like this is the live question. Today, only a hyperscaler can, because a mixed fleet demands three scarce things at once: visibility into your own workloads, a second software stack, and capital. All three are getting cheaper. If the serving and storage layers keep abstracting the hardware, the tier below the hyperscalers inherits hyperscaler procurement economics, and the 2029 building gets built by companies that do not exist yet. If the abstraction stalls, the future is smaller: five private fleets, and a merchant market that still shops for the one best plane. Watching where those layers land will tell you more about 2028 silicon share than any chip roadmap.
What Would Make This Wrong
Three ways. NVIDIA absorbs the heterogeneity in-house, the way Rubin plus the Groq rack already gestures, and the fleet diversifies without the vendor list changing. Model architecture shrinks state faster than context grows it, latent attention’s 93% cache cut being the proof such compression is possible, and the old FLOPS logic claws territory back. Or the second software stack stays expensive enough that everyone below the top five rationally pays the CUDA premium forever. Each of these slows the fragmentation. None reverses it while decode stays memory-bound, and decode is where the tokens come from.
The Scoreboard
The framework compresses into a table: name the workload, find its binding constraint, and the winner falls out.
Three calls to hold me to. By end-2027, at least two of the five largest AI buyers serve one frontier model family across three silicon vendors in production. ASIC-based systems pass 35% of AI server shipments, with the gain coming from inference. And at least one non-hyperscaler inference provider runs a mixed NVIDIA and non-NVIDIA fleet at scale. If all three miss, the thesis was wrong and the one-plane world won.
Inference is becoming stateful. Weights, context, and KV cache accumulate in specific places, and moving that state increasingly costs more than choosing the theoretically fastest chip. So the compute comes to the data, rather than the data forever shuttling to the compute.
Data has gravity. Accelerators fall toward it.