Why AI Is a Storage Workload

The GPU isn’t the bottleneck anymore — data movement is. Inference runs on accumulated state, and that state needs somewhere to live.

Where AI inference state lives: the tier ladder from HBM to object storage, and the flash tiers that just entered the hot path’

Thesis. The most expensive processor in the data center spends most of its time waiting for bytes. Yet the consensus AI-infrastructure stack has three headline layers — GPUs, networking, HBM — and treats storage as plumbing, because of one assumption: training is a storage workload, but inference is stateless. Tokens in, tokens out, nothing to keep. That assumption is now wrong. Stated precisely: inference is becoming a state-management problem — compute increasingly exists to transform state, and storage is where state persists. The clearest evidence is on the API price sheets: OpenAI, Anthropic, and Google sell cached input tokens at ~90% off; DeepSeek, which parks cached context on SSDs, at ~99% off. Those discounts exist because a stored byte of context replaces recomputation — every gigabyte of remembered context is compute you never have to buy. Long context, agents, RAG, and multimodal serving are manufacturing that state at extraordinary rates, and it lands on flash — which is why NAND, a $270B market growing ~281% year-over-year, is repricing faster than any memory cycle on record. The strategically interesting line in AI infrastructure is no longer just the network between GPUs. It is the boundary between memory and storage.

AI storage by the numbers: NAND market size, KV cache per session, cached-token discounts, record eSSD quarter, HDD lead times, Google token volume

The Bottleneck Moved

Start with a fact that should be better known: during generation, a GPU is mostly not computing. Emitting one token means streaming every active model weight and the session’s entire KV cache out of memory — well over 100GB per step for a 70B-class model — and the arithmetic waits on the bytes. The imbalance is quantifiable: at batch size one, decode performs roughly 2 FLOPs for every byte it reads, while a modern accelerator is provisioned at ~300–600 FLOPs per byte of memory bandwidth — the math units are structurally >99% idle unless batching fills them, and batch size is capped by how much KV cache fits in memory. So memory capacity, not FLOPs, increasingly sets cost per token. The weights, meanwhile, are the most-moved bytes in the data center: consumed at full HBM bandwidth, a 140GB model is read end-to-end about 24 times per second — on the order of two million complete passes a day per serving instance. Penguin Solutions summarizes the resulting split: serving is roughly 30% compute, 70% memory. AI infrastructure has four layers — compute, memory, network, storage — and the last three are all versions of the same problem: how fast can you move bytes to the arithmetic? Training answered with HBM and NVLink, and the industry concluded the answer was permanent. But inference moves different bytes — context, weights, adapters, embeddings — and, critically, its bytes outlive the request. State that survives has to land somewhere, and each tier it can land on trades latency for capacity and cost. That is why inference optimization is turning into memory-hierarchy design rather than FLOPs procurement — and why the bottom of the hierarchy, storage, has stopped being a passive archive and become an active participant in serving.

The Blind Spot

Follow the dollars, not the discourse. Nearly all AI-memory coverage is about HBM — a market that even bullish forecasts sized around $55B for 2026, before this year’s repricing. NAND flash, which almost nobody covers, is a $270.6B market in 2026, heading for a projected $379B in 2027 (TrendForce, May 2026). For scale: the entire hard-drive industry books less revenue in a year (~$23B) than the top five NAND vendors now book in a single quarter ($38.9B). Storage stayed out of the AI narrative because the serving story said it could: if inference keeps nothing, storage is just where cold data goes to be cheap. The last eighteen months quietly falsified that premise.

The dollars vs the discourse: 2026 market size for DRAM, NAND, HBM and HDD

Training Was Always a Storage Workload

The training side was never light on storage — it was just a batch problem. MLPerf’s checkpoint benchmark puts numbers on it: a 70B-parameter model writes a 912GB checkpoint; a 1T model, 15TB. Cadence scales with failure rate — Meta’s Llama 3 run logged 419 interruptions in 54 days on 16,000 GPUs, and at 100,000 accelerators the math implies a checkpoint every ~90 seconds, written in under five seconds: a ~3.6TB/s burst. DeepSeek open-sourced the machinery it built for exactly this — 3FS, a flash-native filesystem sustaining 6.6TiB/s of aggregate read. But checkpoints are periodic, sequential, and predictable: a solved class of problem. What changed in 2025–26 is the other side. Serving stopped being stateless.

Training storage demands: checkpoint size by model scale, from 105GB at 8B parameters to 15TB at 1T

Inference Was Supposed to Be Stateless

The state that broke the assumption is the KV cache. As a model reads your context, it builds a scratchpad of intermediate results — attention keys and values for every token — so that each new word only has to consult the notes, not reread the raw text. Throw the scratchpad away and the model must reprocess the entire conversation from scratch before it can say anything. That scratchpad is big: for a 70B-class model it runs ~0.33MB per token, so a 128K-token session carries ~42GB of state — on top of ~140GB of model weights. Four concurrent long sessions exceed the HBM of any GPU ever shipped. Concurrency multiplies it without limit: VAST Data’s arithmetic for a consumer-scale service puts 100,000 concurrent users at ~45PB of retained context — three orders of magnitude beyond what HBM and DRAM can hold. Solidigm’s framing is the right one: an inference service becomes memory-limited long before it becomes compute-limited.

And the KV cache is only the loudest entry in a growing inventory. Model weights are now hot storage objects — DeepSeek-V3 is 688GB on disk, and serving fleets shuffle them constantly. LoRA adapters, a few hundred megabytes each, get swapped per-request by the thousands. Agent memory persists between sessions by design. Each item is small next to a checkpoint; together, multiplied by users, they are the largest new data category in the data center.

The state inventory: what inference now has to keep, from 200MB LoRA adapters to a 45PB context pool, on a log scale

Agents Make The Storage Problem Worse

Agents converted context from a per-request cost into a durable asset. The best public trace data comes from LMCache’s replay of 739 real Claude Code sessions: on average, a session’s context grows from ~20K to ~115K input tokens over an hour-long conversation — a 5.7x inflation, re-sent on every turn, of which 97% is reusable prefix. At 70B-class density, that one session is ~7GB of state at the start and ~38GB by the end. Now scale it: reasoning-and-tool-use models went from negligible to more than half of all tokens on OpenRouter during 2025, and Kioxia now calls agentic AI “the primary growth driver for NAND demand expansion.” A session that lives for an hour, pauses, and resumes tomorrow is not a cache-in-DRAM problem. It is a storage problem.

Agentic sessions inflate context: median Claude Code session grows from 20K to 115K input tokens in one hour

The Quieter State: Generated Media

Multimodal generation is compounding storage demand from the output side. Every generated video, image, and audio clip becomes a persistent asset the moment it exists — and Western Digital’s case work found each AI-generated video is stored at least seven times across platforms: the creation site, the creator’s copies, and every social network it’s posted to. WD’s summary is the structural point of this whole piece: storage demand is cumulative and persistent, unlike compute demand. That asymmetry is underpriced — compute demand can drop overnight on a single efficiency breakthrough; stored bytes never un-accumulate. A GPU finishes its job and moves on. The bytes stay.

The Market Already Prices State

Here is the mechanism in one sentence: reusing context the model has already seen normally means paying full compute to reread it — unless the scratchpad was saved, in which case you pay a disk read. The price sheets quantify the difference: cached input runs ~10% of fresh across OpenAI, Anthropic, and Google, and ~1% at DeepSeek, whose cache has lived on disk since 2024. The vendor benchmarks — all self-run, so treat the exact magnitudes loosely — point the same direction — WEKA reports 20x faster time-to-first-token at 128K context from flash-resident cache; Mooncake, the architecture behind Kimi, reported 75% more requests served on the same GPUs by pooling cache across DRAM and SSD. Storing a long prefix costs pennies per month; recomputing it costs the same dollars every single time. Once context is reused — and agents guarantee reuse — flash wins the arithmetic.

Cached vs fresh input token pricing across OpenAI, Anthropic, Google and DeepSeek — the market prices stored state at 90-99% off recompute

The Arithmetic of Remembering

Make it concrete. A 50,000-token prefix — a system prompt plus a codebase, or an agent’s accumulated working context — carries ~16.5GB of KV state on a 70B-class model. Rereading it costs ~$0.25 at prevailing fresh-input prices, every single time. Storing it costs ~$1.32 a month at ordinary cloud block-storage rates, and serving it back costs ~$0.025 per reuse at cached rates. The breakeven is six reuses a month. An agent reuses its context on every turn — it clears that bar within its first few minutes. At a thousand reuses a month, the choice is $250 of recompute against $26 total, of which the storage itself is $1.32. That asymmetry is why the discounts exist, why every serving stack now tiers to flash, and why “stateless” is ending: remembering costs an order of magnitude less than rethinking.

The arithmetic of remembering: monthly cost of recomputing a 50K-token prefix vs storing its KV cache on flash — breakeven at ~6 reuses per month

Where the State Lands

So does storage actually enter the inference hot path? Yes — through one door. An LLM answers in two phases: prefill (ingest the whole context — compute-heavy) and decode (emit tokens one at a time — memory-bandwidth-heavy). Decode stays welded to HBM and is not leaving. Prefill is the door, because a persistent KV cache makes most of it unnecessary — and NVIDIA has now formalized where that cache lives. Its Dynamo serving stack names the tiers like memory levels: G1 (HBM) → G2 (DRAM) → G3 (local SSD) → G4 (networked storage). At CES 2026 it went further, announcing BlueField-4-based “inference context memory” systems — a pod-level flash tier with a claimed 5x tokens per second, launched with a partner list that reads like the entire storage industry: DDN, Dell, HPE, IBM, Pure, VAST, WEKA and five others. The silicon is following: Kioxia’s GP1 series — built on its low-latency XL-FLASH media specifically for KV cache — samples this year at more than 10 million random-read IOPS, and Samsung’s PCIe Gen6 drive loads a 40GB model in 1.4 seconds. NVIDIA’s storage-certification program now defines a dedicated KV-cache I/O profile. When the GPU vendor starts speccing drive workloads, storage has stopped being plumbing.

Two phases, one door: decode stays welded to HBM; prefill is where a stored KV cache replaces recomputation, reloading through Dynamo’s G1-G4 hierarchy

Related: Why Prefill and Decode are Splitting the GPU

The NAND Shock

The commodity market noticed before the narrative did. NAND contract prices rose ~55–60% quarter-over-quarter in Q1 2026 and another 70–75% in Q2 — records, against the low single digits of a normal cycle — and the top five enterprise-SSD vendors booked a record $18.5B quarter, up 86% sequentially, which TrendForce attributes explicitly to AI-agent services. The supply side cannot chase it: 2026 industry capex puts $61.3B into DRAM against just $22.2B into NAND, mostly process migration rather than new wafers. So buyers are locking the future instead — SanDisk holds $42B in contracted revenue, and Kioxia has roughly half of its 2028 output already committed. TrendForce doesn’t expect the tightness to clear before late 2027.

The NAND shock: record quarterly contract price increases through 2026, and top-5 NAND vendor revenue nearly doubling in a quarter

The strangest confirmation is the hard-drive market. Nearline HDDs — the default home of cold data — hit 52+ week lead times, with Western Digital sold out for all of calendar 2026. The shortage actually widened the price gap flash was supposed to be closing: a 30TB QLC SSD cost 4.9x the equivalent HDD in mid-2025 and 22.6x by Q1 2026. By classical storage economics, QLC demand should have stalled. Instead it had its best quarter ever — hyperscalers building AI pipelines couldn’t get HDDs at any price, and paid a 22x per-terabyte premium for density, power (~30% less), and availability. When buyers pay that premium on allocation, the workload has changed underneath the market.

The QLC paradox: the SSD-vs-HDD price gap widened from 4.9x to 22.6x — and QLC demand boomed anyway

The Stack: Who Builds What

The signature of a real category shift is that every layer ships a product for it within the same 18 months. At the media layer, all five NAND makers are racing to 122–245TB QLC drives and purpose-built AI SSDs. At the systems layer, VAST Data raised at $30B in April 2026 on the back of a $1.17B CoreWeave commitment; WEKA’s flash-resident KV cache went GA; DDN — the storage behind xAI’s Colossus — took $300M from Blackstone at $5B; Pure Storage rebranded to Everpure, won a Meta design for its 150–300TB flash modules, and shipped a KV-cache accelerator in June; and NetApp logged ~500 AI wins in one quarter, more than its entire prior fiscal year.

The cloud tells the same story: S3 alone holds 500+ trillion objects, and per IDC, all-flash arrays crossed half of enterprise storage revenue for the first time in Q1 2026. None of this is a stock pitch; it’s a census. When a dozen vendors ship the same new product category inside two years — and NVIDIA writes the reference architecture legitimizing it — the workload is telling you what it needs.

The AI storage stack: who builds what, from NAND media through systems software to cloud object storage and NVIDIA’s platform layer

What Breaks the Thesis

Three real risks, in descending order. First, software keeps compressing the state: DeepSeek-style latent attention needs ~70KB per token against ~330KB for standard 70B-class attention, and KV quantization cuts another 2–4x. If compression outruns context growth, the spill to flash shrinks — though so far context and agent session lengths are growing faster than compression improves. Second, part of the NAND repricing is cyclical: BofA’s pre-melt-up forecast for 2026 NAND growth was +45%, not +281%, and the gap between those numbers is froth that can unwind — TrendForce already sees increases moderating to 10–15% in Q3. A 2027 normalization would deflate the headlines without touching the workload argument, but it would deflate them. Third, decode never touches flash — and if specialized prefill silicon (NVIDIA’s Rubin CPX pointed this direction) makes recomputing context trivially cheap, cached state loses value. Even that risk concedes the larger point, though: prefill economics now matter enough that NVIDIA designs separate silicon around them.

Summary

AI’s first infrastructure repricing was compute; the second was memory; the third is underway in storage, driven by a workload change rather than a shortage alone. Inference is stateful now — 42GB per long session, 5.7x context inflation per agent-hour, 45PB per hundred thousand users — and state that outlives a session must live on flash, because DRAM is too small, HDDs are unavailable, and the breakeven against recompute is roughly six reuses a month. The market has already internalized this in three places: the price sheets (90–99% cached-token discounts), the commodity (NAND up ~281% on record eSSD volumes and multi-year LTAs), and the stack (a KV-cache product from every serious storage vendor, with NVIDIA writing the reference architecture). For investors, the value sits at three tolls along the memory-storage boundary: whoever owns cache placement (NVIDIA and the systems software vendors), whoever supplies the flash it lands on (NAND makers, increasingly under long-term contract while capex stays disciplined), and whoever builds the silicon at the boundary itself (the new class of microsecond-latency AI SSDs). Instead of a watch list, here are three calls to hold me to. First: by end-2027, at least one major Western lab follows DeepSeek past a 95% cached-token discount — the arithmetic above says the economics already allow it. Second: by end-2027, every major cloud sells a named “context memory” tier the way it sells block storage today — NVIDIA’s G-tier taxonomy becomes product SKUs. Third: storage’s share of AI data-center spend roughly doubles by 2028, taken mostly from dollars that would have gone to HDD and cold tiers. If all three miss, the thesis was wrong.

Zoom out far enough and this is the oldest pattern in computing. Every wave — mainframes, databases, the web, cloud — eventually stopped being constrained by processing data and became constrained by moving it. AI looked like the exception because training really is compute-bound. Inference is the regression to the mean. This is the thesis this blog is named for: data has gravity. Compute is light and fungible; accumulated state is heavy and it stays put — compute exists to transform state, storage is state’s home, and silicon, networks, and price sheets all end up bending toward where it sits. Storage spent the first three years of the AI buildout as the boring layer. The state has arrived, and it needs somewhere to live.

This is the foundation essay of a series on data movement as AI’s real constraint. Coming next: the memory stack as the new compute stack (HBM → DDR → CXL → SSD → object storage), the rise of inference engineering, and why every AI company becomes a data infrastructure company.

Data Gravity covers AI infrastructure, compute economics, and durable software systems. Related coverage: Why KV Cache and Memory Drive AI Economics, The AI Memory Stack, and What Is CXL?.