Companies keep making AI cheaper per task, but if everyone uses it far more, does the total bill still go up?
Can inference prices fall faster than AI usage grows?
This explores whether the cost of running AI models is dropping fast enough to keep up with how quickly people and agents are using them, or whether demand is outrunning the savings.
This explores whether falling per-use prices can outpace how fast AI usage is growing, which decides whether AI gets cheaper overall or ends up bottlenecked. The corpus suggests the answer depends on what you measure. The price of a fixed level of capability is falling remarkably fast. The demand for tokens is growing even faster, and much of that demand is for capability that hasn't gotten cheap yet.
Start with the optimistic side. Epoch AI's Denain argues that the cost of reaching a given capability level falls roughly 5–10x per year, through distillation, better algorithms and cheaper hardware. On that view, today's expensive capabilities become affordable within a few years Does inference cost remain a persistent economic burden?. When researchers remove hardware price drops and competitive pricing, algorithmic progress alone still delivers about 3x efficiency gains a year How fast is AI inference actually becoming more efficient?. Some of that comes from unglamorous design choices. Tuning a model's shape for inference rather than training alone produced 42% more throughput at equal or better accuracy Can architecture choices improve inference efficiency without sacrificing accuracy?.
The pessimistic side is about volume. Emberson and Sevilla estimate that token demand is growing about 10x per year, while the physical capacity to serve those tokens grows only about 3.4x. They point to early strain such as Anthropic cutting quotas at peak hours Will AI inference demand outpace supply capacity soon?. The structural reason is that agents change what usage means. A chatbot waits for a person to type, but an autonomous agent consumes inference continuously, which is a growth axis training never had Will inference compute soon exceed training compute demand?. Reasoning models add another multiplier. They win by spending more tokens per answer, and non-reasoning models can't close the gap just by being given more compute Can non-reasoning models catch up with more compute?. Smaller models can match larger ones on hard prompts if they think longer Can inference compute replace scaling up model size?. So the industry is swapping model size for token count, which pushes usage up.
The detail you might not expect is that prices fall fastest right after a capability first appears, about 66% per quarter, then slow to about 32% per quarter within two years. Epoch attributes this to temporary monopoly pricing that competition gradually wears away Do AI performance prices fall faster when capabilities debut?. People mostly want frontier capability, so they keep paying the newest, least-discounted prices, even as last year's capability becomes almost free. There is also a floor. In a 1-gigawatt data center, servers account for about 60% of the cost and energy only about 7%, so serving costs are tied to how fast chips get cheaper, not just to software cleverness What actually drives AI data center costs?.
The most promising route to price beating usage may be to stop wasting tokens rather than make each token cheaper. Giving easy prompts less compute and hard prompts more beats uniform budgets Can we allocate inference compute based on prompt difficulty?. Models can also learn when to think at length and when to answer directly Can models learn when to think versus respond quickly?. At the extreme, a 150M-parameter model that reasons internally without writing out its steps reached a competitive ARC-AGI score at $0.0007 per task Can latent reasoning match chain-of-thought cost efficiency without verbalizing?. The corpus doesn't settle the economics question. It does show that per-unit prices are clearly falling, total demand is probably rising faster, and the outcome depends on whether efficiency per task can keep up with how many tasks agents take on.
Sources 12 notes
Denain argues the inference cost burden is not persistent because the dollar cost to reach a given capability level drops rapidly each year through distillation, algorithmic efficiency gains, and hardware advances. He documents a 3x cost reduction over eight months on a math benchmark and projects 5–10x annual reductions, suggesting today's expensive capabilities become affordable within years.
After removing hardware price declines and competitive pricing effects by analyzing open-weight models, researchers estimate algorithmic progress alone drives roughly 3× annual efficiency gains. This aligns with independent energy efficiency measurements, suggesting the figure is robust across methods.
Augmenting scaling laws with hidden size, MLP-to-attention ratio, and GQA configuration enables architecture optimization for inference. Optimized models achieved up to 2.1% higher accuracy and 42% greater throughput than LLaMA-3.2 under identical training budgets.
Emberson and Sevilla estimate global token demand at 200 million to 4 billion tokens per second, growing 10× yearly, while inference supply grows only 3.4× yearly due to compute bottlenecks. They cite early signs including Anthropic's quota reductions during peak hours.
A seven-layer AI compute stack model reveals that inference demand grows along a new axis training never had: every autonomous agent is a continuous inference consumer. This makes inference, not training, the likely dominant compute demand as agents and robots scale.
Show all 12 sources
Reasoning models persistently outperform non-reasoning models regardless of inference budget because training instills a reasoning protocol that makes additional tokens productive. The gap is fundamentally about deployment mechanisms and training structure, not raw capability.
Snell et al. (2024) showed that inference-time compute trades off against model parameter scaling, especially on difficult prompts. This reveals pretraining and inference compute are not independent resources.
Epoch AI found that across five benchmarks, performance costs drop 66% per quarter when newly state-of-the-art, then slow to 32% per quarter within two years. They attribute this to temporary monopoly pricing on novel capabilities that competition gradually erodes.
Epoch AI's model finds a 1GW data center costs $8.5B annualized, with servers at 60% ($5B/year) versus energy at just $0.6B/year. Capital recovery of hardware dominates operating expenses.
Research shows inference effectiveness varies dramatically by prompt difficulty. Reallocating the same total compute adaptively—giving easy prompts less and hard ones more—substantially outperforms larger models under uniform budgets.
Thinkless trains a single model to select between extended reasoning and direct responses using DeGRPO, which decouples mode selection from answer refinement. This prevents mode collapse and enables self-calibrated routing without explicit difficulty labels.
A 150M-parameter model combining in-context demonstrations with iterative latent computation reached 29.5% pass@2 on ARC-AGI-1 at $0.0007 per task, surpassing previously reported cost-accuracy tradeoffs. The approach separates learning (via demonstrations updating recurrent memory) from reasoning (via iteration in hidden space) without generating intermediate tokens.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Is a compute crunch coming?
- The Price of Progress: Price Performance and the Future of AI
- Reasoning Models Can Be Effective Without Thinking
- How persistent is the inference cost burden?
- When More Thinking Hurts: Overthinking in LLM Test-Time Compute Scaling
- The rising costs of training frontier AI models
- The plunging price of thought
- Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers