How AI Tokens Are Made
Inside the five scheduling, memory, and routing problems that turn GPU compute into output tokens — and why neoclouds keep buying the companies that solved them.
Inspired by a video by Robert from Anyscale — five reasons LLM inference broke the classic model-serving playbook, filmed on 10% battery after the mic died. Worth your 6 minutes and 40 seconds.
Three GPU clouds have now bought inference software instead of building it themselves. The latest deal landed this morning — so before anything else, congratulations to Robert and the entire Anyscale team: Nscale has entered a definitive agreement to acquire Anyscale, the company founded by the creators of Ray. Anyscale keeps its brand, its ~200-person team joins Nscale, and the deal is expected to close in H2 2026. My read: this becomes Nscale’s inference platform strategy. A company that owns power, data centers, and silicon just bought the layer that turns raw compute into tokens.
Sixteen months, three deals, one direction — up the stack:
The thesis behind all three: raw FLOPs are a commodity; delivered tokens per dollar are not. The margin between the two lives in software, because LLM inference is a genuinely different computational problem than the model serving that preceded it. Robert’s video is the cleanest six-minute explanation of why I have seen, and his five differences map neatly onto the life of a single request — how it gets admitted to the hardware (batching), how it executes (prefill and decode), what it remembers (the KV cache), where it lands (routing), and what shape the model underneath even has (sharding). This post works through each with the numbers attached. The list isn’t exhaustive — speculative decoding and quantization are their own posts — but these five are the reasons vLLM, SGLang, and TensorRT-LLM exist as a category, and the reasons that category keeps changing hands.
Difference #1: Variable-Length Computation & Continuous Batching
A ResNet doing image classification is a fixed-shape problem: fixed-size input, fixed FLOP count, fixed-size output. Batching is trivial — stack 64 images, run the graph, return 64 labels. LLMs break both ends of that contract. Prompts vary from 10 tokens to 200K, and outputs vary unpredictably because the model decides when to stop. Batch eight requests naively and the whole batch waits on the longest sequence while finished slots sit idle — you are burning HBM and FLOPs on padding.
The fix is continuous batching (iteration-level scheduling): the engine operates at the granularity of a single decode step, evicting finished sequences and admitting queued ones every iteration, so the batch stays full even as its membership churns. This is the single highest-leverage optimization in LLM serving. Anyscale’s own benchmarks showed up to 23× throughput over naive request-level batching — a vendor number and a best case, so discount it accordingly, but even the peer-reviewed vLLM results attribute 2–4× at equal latency to iteration-level scheduling and the memory management it enables. Either figure has no analogue in fixed-shape model serving. For a GPU cloud, this is the whole P&L in miniature: utilization is the denominator of cost per token, and every point of it recovered by the scheduler drops straight into the margin on delivered tokens.
Difference Two: Prefill Is Compute-Bound, Decode Is Bandwidth-Bound
LLM inference is really two workloads wearing one API. Prefill processes the entire prompt in parallel — thousands of tokens per matmul — and saturates the tensor cores. Decode generates one token at a time, and each step must stream the model weights and KV cache through HBM to produce a handful of FLOPs. The hardware ratio tells you everything: an H100 delivers roughly 989 dense BF16 TFLOPS against 3.35 TB/s of HBM3 bandwidth. Prefill sits comfortably above the roofline’s ridge; small-batch decode sits an order of magnitude below it. Same GPU, opposite bottlenecks.
Schedule both regimes on the same devices and they interfere: a long prefill stalls every decode stream sharing the GPU, which is exactly the inter-token latency jitter users feel. The structural answer is prefill/decode disaggregation — separate compute pools per stage, with the KV cache shipped between them — which lets you size, batch, and even select hardware independently for each regime (compute-dense parts for prefill, bandwidth-rich parts for decode). This is now table stakes in frontier serving stacks, and it is an orchestration problem, not a kernel problem — which is precisely why Ray sits underneath it, and why Anyscale was worth buying.
Difference 3: GPU Memory Management & the KV Cache
The KV cache is where LLM inference stops being a compute problem and becomes a memory problem. In practice, HBM capacity — not FLOPs — is the binding resource. Every attention layer stores key/value vectors for every token in context so they are not recomputed at each decode step, and the sizes are brutal even with grouped-query attention: a Llama-3-70B-class model carries about 320 KB of KV state per token — roughly 10 GB for a single 32K-token conversation (the full arithmetic is in the figure). A handful of long-context sessions can consume more HBM than the weights themselves, and the classic approach of pre-allocating contiguous max-length buffers wasted 60–80% of KV memory to fragmentation and over-reservation in pre-vLLM systems.
The breakthrough Robert references — vLLM’s PagedAttention — imports fifty years of operating-systems thinking: chop the cache into fixed-size blocks, map logical to physical blocks through a table, and share identical prefix blocks across sequences copy-on-write. The vLLM paper reported memory waste falling under 4%, which translated into 2–4× throughput at equal latency simply because more sequences fit in the batch.
Then comes the policy layer: which prefixes to retain in HBM, which to swap to host memory or NVMe, which to recompute. None of this exists in ResNet serving. All of it determines your cost per token.
Difference #4: Prefix-Aware Routing
Scaling classic model serving is a solved problem: replicate the model, put a load balancer in front, route round-robin or least-loaded. That works because every replica is interchangeable — no replica holds state that makes it a better destination for a given request. KV caching destroys that symmetry. Once replica 2 holds the cached prefix for your multi-turn conversation (or your agent’s 15K-token system prompt), sending turn six to replica 3 forces a full re-prefill of state that already exists in the cluster. The longer the shared prefix, the worse the miss: at agentic prompt lengths, prefill dominates time-to-first-token, so a cache hit removes most of the wait — routinely an order of magnitude on TTFT.
Prefix-aware routing makes the load balancer cache-conscious: hash the prompt prefix, track which replica holds which blocks, and trade off cache affinity against load. This matters more every quarter, because workloads are shifting toward exactly the shapes that reward it — multi-turn agents, shared system prompts, RAG templates — where the majority of input tokens are re-reads of tokens the cluster has already processed. It is also why the router is migrating from dumb infrastructure into the inference engine itself: the scheduler now needs a cluster-wide map of KV state to make a good decision. Routing became stateful, and stateful routing is software the neoclouds did not have.
Difference #5: Model Sharding for Mixture-of-Experts
The last break with the old playbook is the model’s own shape. Classic scale-out replicates the whole network on every GPU: N devices, N identical replicas. Frontier models made that impossible twice over. First they outgrew a single device. Then they went mixture-of-experts — attention layers interleaved with expert layers containing dozens to hundreds of feed-forward experts, of which only a few activate per token. DeepSeek-V3 is the canonical public example: 671B total parameters, ~37B active per token across 256 routed experts per MoE layer. The economics are the point — you buy frontier quality at a fraction of the per-token FLOPs — but only if the serving topology cooperates.
The standard deployment replicates attention across GPUs while sharding the experts between them (expert parallelism), then routes each token mid-forward-pass to whichever device holds its activated experts. As Robert puts it, you no longer have ten independent replicas — you have one big replica with a routing fabric inside it. Every decode step becomes an all-to-all communication event, expert load-balancing becomes a live scheduling problem, and the failure domain is the whole pod rather than one box. Since most frontier models are now MoE, this is the default serving problem, not the exotic one — and it is squarely an engine-plus-orchestration problem of the kind Ray was built for.
These five properties compound into one conclusion: the inference engine is a real layer of the stack — as distinct from the GPU below it as the database was from the disk.
The Data Gravity Take
Pull the five threads together and the point is compounding. Continuous batching multiplies with paged memory, which multiplies with cache-aware routing, disaggregation, and expert-parallel sharding — none of these moves cost per token by 5%, and together they move it by integer multiples. That stack is the difference between a GPU cloud earning commodity rental yields and one earning platform margins on delivered tokens, and from there the M&A wave explains itself: CoreWeave took the developer workflow with Weights & Biases, Nebius took the optimization stack with Eigen AI for ~$643M, and today Nscale took Anyscale.
The obvious objection: vLLM and SGLang are open source, so what exactly is anyone paying $643M for? The answer is that the free part is the smaller part. An engine runs a model well on a node — that is where open source has already won. Everything above the node is not free at all: the cluster-wide KV map that makes routing decisions, the disaggregated prefill and decode pools, the autoscaling across heterogeneous fleets, the multi-tenant scheduling, and the small number of teams on earth who have operated all of it at production scale. That is the orchestration layer, and it is what these deals actually price. It is no accident that all three targets sit there rather than at the kernel level — and no accident that Nscale bought the company whose founders created Ray, the substrate the industry already uses to wire these five techniques together, rather than simply adopting an engine anyone can download.
For founders, the lesson is that the value line has moved up-stack: differentiation in inference now lives in schedulers, memory managers, and routers, not in access to silicon. For investors, watch the remaining independents in the orchestration layer — the strategic buyers have shown their hand three times in sixteen months, and every vertically integrating compute platform still missing this layer is a motivated acquirer. Gravity, as ever, pulls the software toward the compute.
Thanks again to Robert and the Anyscale team — for the video, for Ray, and for a decade of making distributed AI legible. Congratulations on the next chapter.