Why is the memory chip sitting right next to AI processors getting so much pricier, so fast?
What is driving the price increase in high-bandwidth memory supplies?
This explores why high-bandwidth memory (HBM), the fast memory stacked next to AI chips, keeps getting more expensive, and what in the wider AI economy is pushing that price up.
This explores why high-bandwidth memory (HBM), the fast memory stacked beside AI accelerators, has been getting more expensive. One caveat first: the collection doesn't cover the manufacturing side of HBM, such as fab capacity or supplier concentration. What it does show is how big the price shift is and the demand pressures around it. The clearest data point comes from Epoch AI's component cost model. HBM's share of AI chip spending rose from 52% to 63% between early 2024 and late 2025, and HBM spending nearly tripled in dollar terms. Over the same period, spending on the logic (the compute part of the chip) stayed flat, and packaging costs fell Is memory becoming the dominant cost in AI chips?. The model attributes this to two things together: higher prices and tight supply. In other words, memory, not compute, is where the money in an AI chip is now going.
Why is demand pushing so hard on memory specifically? Part of the answer is that a lot of AI work is limited by memory speed rather than raw compute. A small but telling example comes from mobile hardware. MobileLLM found it was faster to *recompute* a transformer block twice than to fetch a second set of weights from memory Does recomputing weights cost less than moving them on mobile?. When moving data costs more than calculating with it, the fastest memory becomes the scarce input. The research side points the same way. One synthesis argues that memory architecture has overtaken parameter count as the main dimension along which models now scale Has memory architecture replaced parameter count as the scaling frontier?. If the field's best gains come from restructuring memory, hardware demand follows.
The second pressure is the growth in inference demand. Emberson and Sevilla estimate that global token demand is growing about 10× a year, while inference supply grows only about 3.4× a year. They point to early signs of a 'compute crunch', such as Anthropic cutting usage quotas during peak hours Will AI inference demand outpace supply capacity soon?. Reasoning models make this worse. Spending more compute at inference time can substitute for a bigger model Can inference compute replace scaling up model size?, and frontier models' absolute running costs are rising 3–18× a year even as performance per dollar improves Is benchmark progress outpacing the cost to achieve it?. Long reasoning chains and persistent agent sessions keep large amounts of context in memory. In one 115-day agent case study, 82.9% of tokens were cache reads Do persistent agents really cost less per token?. Context that has to be held and reread drives demand for exactly the memory HBM provides.
This matters beyond chip pricing because hardware dominates AI economics. Epoch AI's model of a one-gigawatt data center puts servers at about 60% of the $8.5B annualized cost, while energy is only about $0.6B a year What actually drives AI data center costs?. So a memory price spike passes straight into the cost of running AI. That creates an interesting tension. Epoch AI's Denain argues that the cost of reaching a given capability level falls 5–10× a year Does inference cost remain a persistent economic burden?, and new capabilities get cheaper fastest right after they debut Do AI performance prices fall faster when capabilities debut?. Rising HBM prices are one place where that falling cost curve meets a physical supply limit. The thing you may not have expected to learn: the AI cost squeeze is increasingly about how fast data can be moved, not how much arithmetic chips can do. That is why one memory component now takes most of the chip budget.
Sources 10 notes
Epoch AI's component cost model shows HBM's share of total AI chip spending rose 11 percentage points between Q1 2024 and Q4 2025, driven by price increases and supply tightness. HBM spend nearly tripled in dollar terms while logic held flat and packaging and auxiliary components declined.
MobileLLM shows that on memory-bound mobile hardware, sharing weights between adjacent transformer blocks by recomputing one block twice uses less latency than fetching separate weights, gaining accuracy with no parameter increase.
Three converging signals in late-2025 research—taxonomy maturation, memory-aware test-time scaling loops, and hybrid sparsity laws—show that returns from restructuring memory now exceed returns from adding parameters. The design bottleneck has shifted from compute to memory structure.
Emberson and Sevilla estimate global token demand at 200 million to 4 billion tokens per second, growing 10× yearly, while inference supply grows only 3.4× yearly due to compute bottlenecks. They cite early signs including Anthropic's quota reductions during peak hours.
Snell et al. (2024) showed that inference-time compute trades off against model parameter scaling, especially on difficult prompts. This reveals pretraining and inference compute are not independent resources.
Show all 10 sources
Regression analysis of frontier model pricing shows benchmark performance per dollar improves 5–10× annually, yet the absolute cost to run these models rises 3–18× annually. Roughly half of measured progress on GPQA-Diamond tracks increased inference spending rather than capability gains independent of compute.
A 115-day case study found 82.9% of tokens were cache reads. When context persists and reuses, the meaningful cost denominator becomes completed artifacts, not individual tokens.
Epoch AI's model finds a 1GW data center costs $8.5B annualized, with servers at 60% ($5B/year) versus energy at just $0.6B/year. Capital recovery of hardware dominates operating expenses.
Denain argues the inference cost burden is not persistent because the dollar cost to reach a given capability level drops rapidly each year through distillation, algorithmic efficiency gains, and hardware advances. He documents a 3x cost reduction over eight months on a math benchmark and projects 5–10x annual reductions, suggesting today's expensive capabilities become affordable within years.
Epoch AI found that across five benchmarks, performance costs drop 66% per quarter when newly state-of-the-art, then slow to 32% per quarter within two years. They attribute this to temporary monopoly pricing on novel capabilities that competition gradually erodes.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- The Price of Progress: Price Performance and the Future of AI
- The rising costs of training frontier AI models
- Is a compute crunch coming?
- How persistent is the inference cost burden?
- The plunging price of thought
- AI chip component costs: memory at 63%
- Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs
- Reasoning Models Can Be Effective Without Thinking