Can tweaking a model's internal shape, not just its size, make it run faster and smarter for free?
How do architectural changes affect inference efficiency independent of model size?
This explores whether the *shape* of a model, meaning how its parameters are arranged and how its reasoning is routed, can make it cheaper or faster to run without simply making it smaller.
This explores whether the shape of a model, and not just its size, determines how cheaply it runs. The corpus says yes, and goes further: at a fixed parameter count, architecture can buy speed and accuracy together. The clearest evidence is work that adds architectural variables to scaling laws Can architecture choices improve inference efficiency without sacrificing accuracy?. Classic scaling laws mostly track parameter count and training data. Once you also account for hidden size, the ratio of feed-forward (MLP) layers to attention layers, and how attention heads share memory (grouped-query attention, or GQA), you can design models that run up to 42% faster than LLaMA-3.2 on the same training budget and also score slightly higher. Two models with the same parameter count can cost very different amounts to serve.
Arrangement matters for quality too. At small scales, MobileLLM found that deep, thin models beat wide, shallow ones of the same size by a few accuracy points Does depth matter more than width for tiny language models?. This contradicts the older view that shape barely matters. That result is about getting more from each parameter, not raw speed, but for on-device models it amounts to the same thing: more quality from the same memory footprint.
An adjacent finding is that many of the biggest inference savings come from changing *how the model spends its computation* rather than what it is built from. LOTUS loops a transformer over its own internal states instead of writing out chain-of-thought one token at a time. It matches explicit reasoning on grade-school math while cutting thinking time by 2.5–6.9× Can latent reasoning close the scaling gap with explicit chain-of-thought?. Thinkless trains one model to decide when a question deserves extended reasoning and when a quick answer will do Can models learn when to think versus respond quickly?. Split designs like SoftCoT keep the large model frozen and hand the 'thinking' to a small helper model Can continuous reasoning avoid forgetting in instruction-tuned models?. Even the input can be redesigned: LLMLingua uses a small model to compress prompts by up to 20× with little accuracy loss Can prompts be compressed without losing model performance?.
This ties into a bigger shift. Model size and inference compute turn out to be interchangeable to some degree: a smaller model given more compute on hard prompts can match a larger one Can inference compute replace scaling up model size?, especially if that compute is spent adaptively by difficulty Can we allocate inference compute based on prompt difficulty?. But the extra compute only pays off if the model was trained to use it. Non-reasoning models never catch up with reasoning models, however much inference budget they get Can non-reasoning models catch up with more compute?. So 'architecture' in the broad sense covers the training regime and routing logic, not just layer counts. Together, these levers help explain why the cost of reaching a given capability is falling roughly 5–10× a year Does inference cost remain a persistent economic burden?.
One gap to flag: these retrievals cover GQA and depth versus width, but not mixture-of-experts, state-space models, or sparse attention. The strictly 'hardware-level architecture' side of this question is thinner in this slice of the collection than the reasoning-efficiency side.
Sources 10 notes
Augmenting scaling laws with hidden size, MLP-to-attention ratio, and GQA configuration enables architecture optimization for inference. Optimized models achieved up to 2.1% higher accuracy and 42% greater throughput than LLaMA-3.2 under identical training budgets.
MobileLLM shows deep-and-thin architectures yield 2.7–4.3% accuracy gains over balanced designs at 125M–350M scale by composing abstract concepts through layers rather than spreading parameters across width.
LOTUS, a looped Transformer trained with gold CoT supervision at each latent position, closes the latent-to-explicit gap on GSM8K and cuts thought-phase latency by 2.5× to 6.9×. Latent reasoning need not be verbalized token-by-token to remain legible and competitive.
Thinkless trains a single model to select between extended reasoning and direct responses using DeGRPO, which decouples mode selection from answer refinement. This prevents mode collapse and enables self-calibrated routing without explicit difficulty labels.
SoftCoT avoids catastrophic forgetting by keeping the main LLM frozen while delegating soft thought generation to a small auxiliary model. This architectural separation maintains pre-trained knowledge while enabling continuous reasoning.
Show all 10 sources
LLMLingua achieves up to 20x prompt compression with only 1.5-point performance drops by allocating less compression to instructions and questions, more to demonstrations, then pruning tokens using a small model's perplexity scoring aligned to the target LLM.
Snell et al. (2024) showed that inference-time compute trades off against model parameter scaling, especially on difficult prompts. This reveals pretraining and inference compute are not independent resources.
Research shows inference effectiveness varies dramatically by prompt difficulty. Reallocating the same total compute adaptively—giving easy prompts less and hard ones more—substantially outperforms larger models under uniform budgets.
Reasoning models persistently outperform non-reasoning models regardless of inference budget because training instills a reasoning protocol that makes additional tokens productive. The gap is fundamentally about deployment mechanisms and training structure, not raw capability.
Denain argues the inference cost burden is not persistent because the dollar cost to reach a given capability level drops rapidly each year through distillation, algorithmic efficiency gains, and hardware advances. He documents a 3x cost reduction over eight months on a math benchmark and projects 5–10x annual reductions, suggesting today's expensive capabilities become affordable within years.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers
- Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs
- Reasoning Models Can Be Effective Without Thinking
- When More Thinking Hurts: Overthinking in LLM Test-Time Compute Scaling
- Inference-Aware Prompt Optimization for Aligning Black-Box Large Language Models
- SoftCoT: Soft Chain-of-Thought for Efficient Reasoning with LLMs
- LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity