AI chips keep shifting cost from processors to memory chips — is that a lasting shift, or just a 2024–2025 price spike?
Will memory's share of chip costs continue rising through 2026?
This explores whether high-bandwidth memory (the fast, stacked memory sitting next to an AI chip's processor) will keep taking a bigger share of what AI chips cost to build, or whether the 2024–2025 surge was temporary.
This explores whether memory's growing share of AI chip costs is a lasting trend or a passing spike. The corpus can't settle this directly: its most recent hard number stops at the end of 2025, and nothing in it forecasts 2026. It does show what pushed memory's share up, and those causes make it possible to reason about whether the rise continues.
The core data point comes from Epoch AI. High-bandwidth memory (HBM) went from 52% to 63% of AI chip component spending between early 2024 and late 2025. Spending on HBM nearly tripled in dollars, while spending on the logic (the processor itself) stayed flat and packaging costs actually fell Is memory becoming the dominant cost in AI chips?. The key detail is *why*: rising prices and tight supply, not just chips carrying more memory. That gives two different futures. If the share rose mainly because memory makers could charge more during a shortage, it could level off or reverse as production catches up. Epoch's separate pricing work fits this pattern: prices for AI capabilities fall fastest right after they debut, because early scarcity pricing gets eroded by competition Do AI performance prices fall faster when capabilities debut?. That paper is about AI services, not memory chips, but the logic of temporary scarcity premiums carries over.
The demand side points the other way. Researchers increasingly see memory, not raw computing power, as the bottleneck worth engineering around. One synthesis argues that memory architecture has replaced parameter count as the main scaling frontier Has memory architecture replaced parameter count as the scaling frontier?, and hybrid designs that pair fast memory lookup with computation are beating designs that use computation alone Can lookup memory and computation work together better than either alone?. Even on phones, moving model weights in and out of memory is so expensive that it's cheaper to *recompute* a block of the model twice than to fetch separate weights Does recomputing weights cost less than moving them on mobile?. When moving data costs more than calculating, buyers keep paying for memory bandwidth.
Inference demand adds pressure too. Benchmark performance per dollar improves 5–10x a year, but the absolute cost of running frontier models rises 3–18x a year, because more of the progress comes from spending more compute at answer time Is benchmark progress outpacing the cost to achieve it?. Generating long answers depends heavily on memory bandwidth, so this trend favors memory spending. There's an important caveat, though: one study finds the long-context bottleneck is really compute (the work of turning old context into the model's internal state), not memory capacity Is long-context bottleneck really about memory or compute?. So not every memory-sounding problem is solved by buying more HBM.
Why does this matter beyond chip makers? Servers make up about 60% of a one-gigawatt data center's annual cost, far more than electricity What actually drives AI data center costs?. Whatever happens to the memory share of chip costs therefore flows almost directly into what AI costs to run. The honest answer is that structural demand favors memory's share holding or rising, while the supply-driven part of the 2024–25 jump is the piece most likely to fade. The corpus has no 2026 numbers to say which force is winning.
Sources 8 notes
Epoch AI's component cost model shows HBM's share of total AI chip spending rose 11 percentage points between Q1 2024 and Q4 2025, driven by price increases and supply tightness. HBM spend nearly tripled in dollar terms while logic held flat and packaging and auxiliary components declined.
Epoch AI found that across five benchmarks, performance costs drop 66% per quarter when newly state-of-the-art, then slow to 32% per quarter within two years. They attribute this to temporary monopoly pricing on novel capabilities that competition gradually erodes.
Three converging signals in late-2025 research—taxonomy maturation, memory-aware test-time scaling loops, and hybrid sparsity laws—show that returns from restructuring memory now exceed returns from adding parameters. The design bottleneck has shifted from compute to memory structure.
Engram combines O(1) N-gram lookup with Mixture-of-Experts routing, revealing a U-shaped scaling law where balanced allocation to both mechanisms outperforms either alone. Gains appear largest in reasoning and code rather than pure retrieval.
MobileLLM shows that on memory-bound mobile hardware, sharing weights between adjacent transformer blocks by recomputing one block twice uses less latency than fetching separate weights, gaining accuracy with no parameter increase.
Show all 8 sources
Regression analysis of frontier model pricing shows benchmark performance per dollar improves 5–10× annually, yet the absolute cost to run these models rises 3–18× annually. Roughly half of measured progress on GPQA-Diamond tracks increased inference spending rather than capability gains independent of compute.
Research shows the bottleneck is not memory capacity but the compute required to consolidate evicted context into fast weights during offline sleep phases. Performance improves with more consolidation passes, following a test-time scaling pattern on harder reasoning tasks.
Epoch AI's model finds a 1GW data center costs $8.5B annualized, with servers at 60% ($5B/year) versus energy at just $0.6B/year. Capital recovery of hardware dominates operating expenses.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- The Price of Progress: Price Performance and the Future of AI
- The rising costs of training frontier AI models
- The plunging price of thought
- AI chip component costs: memory at 63%
- How persistent is the inference cost burden?
- Key Questions on Energy and AI
- MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases
- Is a compute crunch coming?