Strip out cheaper chips and price wars — how much does AI inference really get cheaper each year just from smarter methods?
How much inference efficiency improves per year after controlling for hardware advances?
This explores how fast AI inference gets cheaper because of better methods (smarter models, better software, distillation), once you remove the share of the savings that comes from cheaper, faster chips.
This explores how much of the falling cost of running AI comes from better methods rather than better hardware. The corpus's most direct answer: about 3x per year. Researchers got this figure by studying open-weight models, which let them strip out two things that muddy the raw price numbers. One is hardware getting cheaper. The other is providers cutting prices to compete. What's left is algorithmic progress, and it comes out at roughly 3x a year. The estimate also matches separate measurements of energy efficiency, which suggests it holds up even when you measure a different way How fast is AI inference actually becoming more efficient?.
That 3x makes more sense next to the headline figure people usually quote. Epoch AI's Denain estimates that the cost of reaching a fixed level of capability falls 5–10x a year. That number bundles together distillation, better algorithms and better hardware Does inference cost remain a persistent economic burden?. Read side by side, the two figures suggest that algorithms account for a large share of the yearly drop, though not all of it. Hardware and pricing make up the rest. Denain's point is that expensive capabilities don't stay expensive for long.
The surprise is what the 3x doesn't solve. Emberson and Sevilla estimate that demand for AI-generated text (counted in tokens) is growing about 10x a year, while the capacity to serve it grows only about 3.4x. They argue a compute crunch is close, and point to early signs like usage limits being cut at peak hours Will AI inference demand outpace supply capacity soon?. So efficiency gains that sound huge on their own can still lose to demand. Cheaper inference doesn't mean abundant inference.
If you want to see where the algorithmic gains come from, the corpus has several examples. Choosing model shapes with inference in mind can raise throughput by 42% at the same training budget Can architecture choices improve inference efficiency without sacrificing accuracy?. Giving easy prompts less compute and hard prompts more beats spending the same amount on every prompt Can we allocate inference compute based on prompt difficulty?. Checking confidence at each reasoning step lets a model drop bad reasoning attempts early, so it gets similar accuracy from fewer attempts Does step-level confidence outperform global averaging for trace filtering?. Mixing simple lookup memory with expert routing is another efficiency lever Can lookup memory and computation work together better than either alone?.
One caveat: 'efficiency' assumes you're buying the same output for less, and that isn't always true. Spending more compute at inference time can make a small model as good as a larger one on hard prompts Can inference compute replace scaling up model size?. But a model without reasoning training can't catch up just by being given more tokens Can non-reasoning models catch up with more compute?. Some of what looks like an inference gain actually comes from training. The corpus has one solid number (about 3x a year) and some context around it, but it doesn't track that figure over time or split it across specific techniques.
Sources 9 notes
After removing hardware price declines and competitive pricing effects by analyzing open-weight models, researchers estimate algorithmic progress alone drives roughly 3× annual efficiency gains. This aligns with independent energy efficiency measurements, suggesting the figure is robust across methods.
Denain argues the inference cost burden is not persistent because the dollar cost to reach a given capability level drops rapidly each year through distillation, algorithmic efficiency gains, and hardware advances. He documents a 3x cost reduction over eight months on a math benchmark and projects 5–10x annual reductions, suggesting today's expensive capabilities become affordable within years.
Emberson and Sevilla estimate global token demand at 200 million to 4 billion tokens per second, growing 10× yearly, while inference supply grows only 3.4× yearly due to compute bottlenecks. They cite early signs including Anthropic's quota reductions during peak hours.
Augmenting scaling laws with hidden size, MLP-to-attention ratio, and GQA configuration enables architecture optimization for inference. Optimized models achieved up to 2.1% higher accuracy and 42% greater throughput than LLaMA-3.2 under identical training budgets.
Research shows inference effectiveness varies dramatically by prompt difficulty. Reallocating the same total compute adaptively—giving easy prompts less and hard ones more—substantially outperforms larger models under uniform budgets.
Show all 9 sources
Local step-level confidence catches reasoning breakdowns that global averaging masks and enables early stopping before traces complete. This approach achieves comparable accuracy gains to naive majority voting with far fewer generated traces, proving trace quality matters more than quantity.
Engram combines O(1) N-gram lookup with Mixture-of-Experts routing, revealing a U-shaped scaling law where balanced allocation to both mechanisms outperforms either alone. Gains appear largest in reasoning and code rather than pure retrieval.
Snell et al. (2024) showed that inference-time compute trades off against model parameter scaling, especially on difficult prompts. This reveals pretraining and inference compute are not independent resources.
Reasoning models persistently outperform non-reasoning models regardless of inference budget because training instills a reasoning protocol that makes additional tokens productive. The gap is fundamentally about deployment mechanisms and training structure, not raw capability.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- When More Thinking Hurts: Overthinking in LLM Test-Time Compute Scaling
- Reasoning Models Can Be Effective Without Thinking
- How persistent is the inference cost burden?
- Is a compute crunch coming?
- The Price of Progress: Price Performance and the Future of AI
- Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs
- Think Twice: Enhancing LLM Reasoning by Scaling Multi-round Test-time Thinking
- SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training