As AI models get used billions of times, does it pay to train a smaller model longer so it's cheaper to run?
What happens to the optimal model size as projected inference volume grows?
This explores how the 'right' size for a language model changes once you count the cost of serving it to millions or billions of users, not only the cost of training it, and what else in the corpus pushes toward or against smaller models.
This explores how the 'right' size for a language model changes once you count the cost of running it for every user request, not only the one-time cost of training it. The short answer is that the optimum shrinks. The well-known Chinchilla recipe asks one question: for a fixed training budget, how big should the model be? When projected inference requests are added to that calculation, teams expecting around a billion requests should train smaller models for longer than Chinchilla recommends Should LLM teams train smaller models for longer?. The extra training looks wasteful on paper, but every request is served by a smaller and cheaper model, so it pays for itself once enough requests come in. The more a model will be used, the more it is worth spending upfront to make it small.
The corpus suggests this is part of a broader shift: model size is no longer the only lever for capability, so shrinking it costs less than you might expect. A smaller model given more compute at answer time can match a larger one on hard prompts Can inference compute replace scaling up model size?. This works even better when that compute is spread by difficulty, with less on easy prompts and more on hard ones Can we allocate inference compute based on prompt difficulty?. The model's shape matters too, not just its parameter count. Choosing hidden size, the balance between MLP and attention layers, and how attention heads are grouped with inference in mind gave 42% more throughput and slightly better accuracy for the same training budget Can architecture choices improve inference efficiency without sacrificing accuracy?. Other notes find capabilities that don't need extra parameters at all. Looking facts up with tools removes the limit on how many facts a model can store in its weights Can models store unlimited facts without growing larger?. Models under 2B parameters can learn to ground their answers in source text when the training curriculum is designed for it Can small models learn to ground answers in context?. A stronger model can even build test-time scaffolding that nearly doubles a weaker model's performance without retraining it Can a stronger model lift a weaker one at test time without retraining?.
There are limits, though. Inference compute can't stand in for everything training provides. Models trained to reason keep beating non-reasoning models however many extra tokens the latter get, because training is what makes those extra tokens useful Can non-reasoning models catch up with more compute?. Shrinking also has a hidden cost on rare tasks. Larger models learn rare skills better, not because they can represent more, but because spare capacity keeps common tasks from overwriting slowly forming rare-task features Why do larger models learn rare tasks better?. A small, heavily trained model may lose exactly those long-tail skills unless the training data mix is designed to protect them.
One more twist complicates the picture. The case for small, overtrained models assumes inference stays expensive. Epoch AI's Denain argues the cost of reaching a given capability level falls roughly 5–10x a year through distillation, better algorithms, and hardware Does inference cost remain a persistent economic burden?. That means the inference volume you project today is weighed against a per-request cost that is itself shrinking. The underlying pattern still holds: the more a model will be used, the smaller it should be. But the corpus doesn't settle how strong that pressure stays as serving costs keep falling, and that is an open question worth following.
Sources 10 notes
Incorporating projected inference requests into Chinchilla scaling laws shows teams expecting ~1B requests should train smaller models for longer than Chinchilla-optimal. The extra training compute pays off once inference volume is high enough.
Snell et al. (2024) showed that inference-time compute trades off against model parameter scaling, especially on difficult prompts. This reveals pretraining and inference compute are not independent resources.
Research shows inference effectiveness varies dramatically by prompt difficulty. Reallocating the same total compute adaptively—giving easy prompts less and hard ones more—substantially outperforms larger models under uniform budgets.
Augmenting scaling laws with hidden size, MLP-to-attention ratio, and GQA configuration enables architecture optimization for inference. Optimized models achieved up to 2.1% higher accuracy and 42% greater throughput than LLaMA-3.2 under identical training budgets.
A formal proof and experiments show in-weight memorization is bounded by model size, while tool-use enables unbounded factual recall through a simple circuit. In-weight finetuning also degrades general capability by overwriting prior knowledge.
Show all 10 sources
Sub-2B models trained on synthetic multi-hop QA can ground answers in passages, cite literal quotes, and abstain from confabulation. The OCC-RAG work shows faithfulness emerges from training curriculum design, not parameter count.
A stronger model built inference-time harnesses that nearly doubled weaker model performance on Theory-of-Mind benchmarks without retraining, primarily by moving unstable reasoning into deterministic code and task-specific routing rather than encouraging extended reasoning.
Reasoning models persistently outperform non-reasoning models regardless of inference budget because training instills a reasoning protocol that makes additional tokens productive. The gap is fundamentally about deployment mechanisms and training structure, not raw capability.
Larger models succeed at rare tasks not because they can represent solutions smaller models cannot, but because abundant capacity weakens gradients on common tasks, preventing them from overwriting slowly-accumulating rare-task features. Data-mixture design may be cheaper than scaling.
Denain argues the inference cost burden is not persistent because the dollar cost to reach a given capability level drops rapidly each year through distillation, algorithmic efficiency gains, and hardware advances. He documents a 3x cost reduction over eight months on a math benchmark and projects 5–10x annual reductions, suggesting today's expensive capabilities become affordable within years.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs
- Reasoning Models Can Be Effective Without Thinking
- Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
- ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning
- When More Thinking Hurts: Overthinking in LLM Test-Time Compute Scaling
- Provable Benefits of In-Tool Learning for Large Language Models
- Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws
- How persistent is the inference cost burden?