Can tweaking a model's shape and 'thinking style' make it cheaper to run, without changing how expensive it was to train?
How do architectural choices reduce inference cost independent of training compute?
This explores how the way a model is designed (its shape, how it reuses layers, whether it 'thinks' in words or in hidden states, how it decides how much effort to spend) can make answering a query cheaper, separately from how much compute went into training it.
This explores how design choices make a model cheaper to run, as opposed to cheaper or more expensive to train. The clearest evidence in the corpus that these are separate levers comes from work that adds architecture to scaling laws. Classic scaling laws only ask how big a model is and how much data it saw. Once you also account for hidden size, the ratio of MLP to attention layers, and grouped-query attention (GQA, which shares key/value memory across attention heads), you can design models that are both more accurate and up to 42% faster at inference than LLaMA-3.2 with the same training budget Can architecture choices improve inference efficiency without sacrificing accuracy?. At small scales the model's proportions matter in a similar way: deep, thin models beat wide, shallow ones below a billion parameters, so you get more capability from the same number of weights to serve Does depth matter more than width for tiny language models?.
A second family of ideas gets depth without adding parameters. Looped models run the same layers again and again instead of stacking new ones. They can beat larger standard models on reasoning because repeating a step helps with tracking state, and the point where the loop stops changing tells the model when to stop Can models learn by looping instead of growing larger?. Push this further and the model stops writing out its reasoning altogether. A 150M-parameter model that iterates in its hidden state, rather than producing chain-of-thought tokens, set a new cost-accuracy frontier on ARC-AGI-1 at about $0.0007 per task Can latent reasoning match chain-of-thought cost efficiency without verbalizing?. SoftCoT takes a modular route: a small helper model produces 'soft' continuous thoughts while the large model stays frozen Can continuous reasoning avoid forgetting in instruction-tuned models?. The common thread is that the cheapest reasoning token is the one you never generate. Energy-based transformers come at it from another angle. They treat inference as an optimization the model can run for longer or shorter, and they get more out of each extra unit of inference compute than standard transformers Can energy minimization unlock reasoning without domain-specific training?.
The third lever is deciding how much to spend on each query. Easy prompts don't need what hard prompts need, and giving each prompt a budget that matches its difficulty beats a uniform budget, even against larger models Can we allocate inference compute based on prompt difficulty?. Thinkless builds that decision into the model: one network learns when to reason at length and when to just answer Can models learn when to think versus respond quickly?. The 'architecture' can also sit outside the model. A stronger model wrote inference harnesses that nearly doubled a weaker model's scores, mostly by moving shaky reasoning into deterministic code and routing, not by making the model think longer Can a stronger model lift a weaker one at test time without retraining?.
The part you may not have expected: the corpus suggests 'independent of training' is only partly true. Inference compute and model size trade off against each other, so a small model that thinks harder can match a large one on hard prompts Can inference compute replace scaling up model size?. But extra inference only pays off if training taught the model to use it. Models without reasoning training don't catch up to reasoning models however much inference budget you give them Can non-reasoning models catch up with more compute?. Even Thinkless's routing is something the model learns during training. So the most useful way to see it is that architecture sets the cost of each unit of thinking, and training sets whether that thinking is productive. The cheapest systems get both right.
Sources 11 notes
Augmenting scaling laws with hidden size, MLP-to-attention ratio, and GQA configuration enables architecture optimization for inference. Optimized models achieved up to 2.1% higher accuracy and 42% greater throughput than LLaMA-3.2 under identical training budgets.
MobileLLM shows deep-and-thin architectures yield 2.7–4.3% accuracy gains over balanced designs at 125M–350M scale by composing abstract concepts through layers rather than spreading parameters across width.
Models that re-apply layers in recurrent depth outperform larger feedforward networks on reasoning tasks. This works because recursion enables state tracking and compositional generalization that parameter scaling alone cannot achieve, with convergence signals providing natural halting.
A 150M-parameter model combining in-context demonstrations with iterative latent computation reached 29.5% pass@2 on ARC-AGI-1 at $0.0007 per task, surpassing previously reported cost-accuracy tradeoffs. The approach separates learning (via demonstrations updating recurrent memory) from reasoning (via iteration in hidden space) without generating intermediate tokens.
SoftCoT avoids catastrophic forgetting by keeping the main LLM frozen while delegating soft thought generation to a small auxiliary model. This architectural separation maintains pre-trained knowledge while enabling continuous reasoning.
Show all 11 sources
Energy-Based Transformers assign energy values to input-prediction pairs and use gradient descent minimization for inference, yielding 35% higher training scaling rates and 29% more inference-compute gains than Transformer++, while generalizing better on out-of-distribution data without domain-specific scaffolding.
Research shows inference effectiveness varies dramatically by prompt difficulty. Reallocating the same total compute adaptively—giving easy prompts less and hard ones more—substantially outperforms larger models under uniform budgets.
Thinkless trains a single model to select between extended reasoning and direct responses using DeGRPO, which decouples mode selection from answer refinement. This prevents mode collapse and enables self-calibrated routing without explicit difficulty labels.
A stronger model built inference-time harnesses that nearly doubled weaker model performance on Theory-of-Mind benchmarks without retraining, primarily by moving unstable reasoning into deterministic code and task-specific routing rather than encouraging extended reasoning.
Snell et al. (2024) showed that inference-time compute trades off against model parameter scaling, especially on difficult prompts. This reveals pretraining and inference compute are not independent resources.
Reasoning models persistently outperform non-reasoning models regardless of inference budget because training instills a reasoning protocol that makes additional tokens productive. The gap is fundamentally about deployment mechanisms and training structure, not raw capability.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Hierarchical Reasoning Model
- Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
- Evaluating Theory of Mind in Reasoning Models: Robustness over Reasoning
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
- Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers
- Reasoning Models Can Be Effective Without Thinking
- Rethinking Thinking Tokens: LLMs as Improvement Operators