The AI Memory Stack
AI's memory problem isn't buying more HBM. It's managing a full hierarchy — GPU cache to cold storage — with a different winner at every tier.
TL;DR — the five things you need to know
Compute and memory have decoupled. Over 20 years, peak server compute grew roughly 3x every two years; memory bandwidth grew only 1.6x. Compounded, that’s a ~600x gap. It now decides how AI infrastructure gets built, not raw compute.
HBM is where the profit concentrates. It’s roughly half the manufacturing cost of an NVIDIA B200, and HBM plus packaging together are close to two-thirds of the chip. SK Hynix owns this tier today; Micron is closing fast; Samsung is recovering.
The scarcity cascades down. HBM’s premium pulled so much fab capacity out of ordinary DDR5 that commodity DRAM prices spiked 93–98% in a single quarter in early 2026.
Inference has its own bottleneck now: the KV cache — a model’s running memory of the conversation so far. A single 128K-context request already needs roughly a third of the memory the model’s own weights take up, and it multiplies with every concurrent user.
There is no single “memory stock.” There are structural winners at each tier — memory, packaging, connectivity, and the software that decides what lives where — and they are not the same companies.
Spending on HBM isn’t the same as owning the trade
“Buy more HBM” explains where the money is going right now. It doesn’t explain who gets to keep it.
AI systems are rebuilding the same memory hierarchy every computer has run on since the 1960s: small, fast tiers next to the chip, and progressively larger, cheaper, slower tiers further away. What’s different is scale. A GPU’s own on-chip memory is tens of megabytes. The cluster behind it needs petabytes. The data lake behind that needs exabytes. Every rung of that ladder is now its own market, with its own scarcity and its own winner.
That is the actual investment question — not “how much HBM capacity exists,” but which company owns which rung, and how durable is that ownership.
The memory wall stopped being theoretical
For 20 years, peak server compute grew roughly 3x every two years. Memory bandwidth grew only 1.6x over the same stretch. Compounded, that gap is why “the memory wall” went from an academic phrase to the constraint that has shaped cluster design since around 2023.
Moving data now costs more than computing on it — a DRAM read burns roughly 100x the energy of an on-chip cache read. That’s the physical reason every layer of the AI hardware stack is being redesigned around keeping data close, not just holding more of it. Compute stopped being the scarce resource. Feeding it is the scarce resource now.
The hierarchy, at a glance
Walk the ladder from a GPU’s own memory down to cold storage, and two things happen at once: capacity keeps growing, and speed keeps falling. That trade-off isn’t an engineering limitation to be solved — it’s physics. Nobody is building a memory technology that is simultaneously as fast as on-chip cache, as cheap as object storage, and as large as a data lake — the economics of stacking silicon mean nobody will.
The rest of this piece walks down that ladder and asks the only question that matters for capital: who owns each rung, and why is that ownership hard to take away.
Where the profit sits
HBM is the tier closest to the compute die, and it’s where the money actually concentrates. A B200 GPU carries up to 192GB of HBM3E delivering roughly 8 TB/s of bandwidth — enough that fitting it into the package now costs more than the compute die itself. Independent teardown estimates put HBM at roughly half of a B200’s total manufacturing cost, and HBM plus TSMC’s advanced packaging together near two-thirds of it.
That premium isn’t temporary, and it isn’t arbitrary. Stacking twelve memory dies on top of each other compounds defect risk with every layer, so yield is the real moat: SK Hynix’s proprietary stacking process reportedly reached 80%+ yield well before its competitors, and that lead is the entire reason it holds roughly 55–60% of the HBM market and crossed a $1 trillion valuation in 2026. Micron is closing the gap fast: fiscal Q3 2026 data-center revenue hit $41.5 billion, up 346% year over year. Samsung, after 18 months of failed qualification attempts, is finally shipping at scale.
None of it works without TSMC. Every HBM stack has to be packaged next to the compute die, and TSMC still can’t add that capacity fast enough: 2026 capex guidance was raised to $60–64 billion for it, and TrendForce still pegs the shortfall at about 10% even after that spending. Own HBM, or own the packaging step it depends on, and you own this tier. There isn’t a third way in.
The DRAM Crunch: Nobody priced this in
The scarcity doesn’t stop at HBM. Manufacturers spent two years pulling fab capacity toward HBM’s fatter margins, which starved ordinary DDR5 — the memory in every regular server and PC. Contract prices for plain DDR5 spiked 93–98% in a single quarter in early 2026, pushing total DRAM industry revenue past $97 billion in that same quarter.
The part of the story that hasn’t landed yet
There’s no new winner in this chart — it’s the same three companies collecting a second round of pricing power on the commodity tier they used to treat as an afterthought. Samsung’s memory division posted a $61 billion quarterly profit in the same period its phone division lost money. That contrast is the cleanest illustration anywhere of where leverage in the semiconductor industry currently sits.
CXL: A tier still proving itself
One rung down, CXL memory pooling is a real idea that hasn’t fully arrived yet. The pitch: let a server borrow memory sitting idle elsewhere in the rack instead of buying more. The waste it targets is real — roughly a quarter of provisioned DRAM sits unused at peak load. The catch is latency: pooled memory runs meaningfully slower than memory plugged directly into the server, so it’s a new, distinct tier, not a drop-in replacement for DRAM. Read our What is CXL piece?
It only started shipping in real hyperscale production in November 2025, when Microsoft Azure launched its first CXL-equipped cloud instances on Astera Labs’ controllers. Astera is the clearest public winner at this layer: FY2025 revenue hit $852.5 million, up 115% year over year. Its newest product line — fabric switching for AI racks — is on pace to become its largest by the end of 2026. The bet there isn’t on any particular memory technology. It’s on being the connective tissue between tiers, which is why it wins regardless of which memory generation or GPU architecture ends up on top.
KV Cache: Inference’s real bottleneck
Inference created a new kind of memory demand that barely existed three years ago. When a model answers a question, it keeps a running memory of the conversation so far — the KV cache — so it doesn’t have to recompute everything on every new word. At long context, it adds up fast: a single 128K-context request on a 70-billion-parameter model needs roughly 40GB of cache against the model’s own ~140GB of weights. Multiply that by the dozens of concurrent long-context sessions a production server actually holds, and cache — not the weights — becomes the largest consumer of GPU memory.
Architecture is a real moat here. DeepSeek-V3’s multi-head latent attention holds cache to about 70KB per token, versus 516KB for Llama-3.1-405B under standard attention — a 7.3x gap that’s pure architecture, unrelated to whose chips it runs on.
The rest of the industry is attacking the same problem a different way: moving the cache off the GPU and down the hierarchy, instead of paying to keep all of it in the most expensive memory available.
Moonshot AI's production system for its Kimi model, built this way, reports serving 75% more requests on the same hardware. This is the newest and least settled layer in the whole stack. Whoever builds the default software for deciding what stays on the GPU and what gets pushed down is building the operating system for the entire hierarchy — a bigger and more durable business than any single memory chip, because it sits on top of whichever hardware generation wins next.
Which data needs to live where
Every type of data an AI system touches ends up somewhere on this ladder, and the cost of misplacing it is asymmetric. Put something in HBM that didn’t need to be there, and you’re burning your most expensive tier on data that could have sat one rung down — a direct hit to margin on every inference call.
Who structurally wins
Ownership concentrates differently at each rung of the ladder.
The physical, scarce tiers — HBM and advanced packaging — are closed oligopolies with multi-year qualification cycles. That’s SK Hynix, Micron, and Samsung on memory, and TSMC on packaging. Micron has crossed $1 trillion in market value too, and TSMC sits near $1.86 trillion. Their sold-out capacity is a structural moat, not a cyclical one, and it’s the highest-conviction part of this thesis precisely because it’s already priced in.
The connective tissue between tiers is a separate, earlier bet. Astera Labs is the clearest public example, and the reason to like it is different: it doesn’t have to bet on which memory or GPU generation wins, because every generation still needs a physical link between them.
The placement software — the layer deciding, in real time, what data lives where — is the newest and least settled tier. It compounds independently of the hardware cycle underneath it, which is exactly what makes it worth watching over the next two years.
The counterweight
Every capacity plan on the table assumes today’s memory intensity per token holds roughly steady. If efficient model designs keep cutting that number the way DeepSeek’s already have, the capacity that looks scarce in 2026 could look merely adequate by 2028 — and the CXL pooling case, which depends on realistic workload economics rather than switch-cost math, could stay a narrower opportunity than the current enthusiasm implies.
The winners criteria
The AI memory stack isn’t a single trade. It’s a ladder, and the money is spreading across every rung of it rather than pooling in one place.
Own a physically scarce tier with real qualification barriers — HBM and advanced packaging reward incumbents, not new entrants.
Own the connective tissue, not the memory itself — durable regardless of which hardware generation wins.
Own the software that decides placement — the newest layer, and the one most likely to compound independently of the silicon cycle.
Discount the assumption that memory intensity per token stays flat — the companies making memory less necessary are the underpriced counterweight to the companies making more of it.
Compute got its platform shift years ago. Memory just got its own — and it’s not one trade, it’s an entire supply chain with a different winner at every tier.