Should an AI think the same amount about every question, or spend more effort only when a problem is actually hard?
How does compute budget allocation work per prompt difficulty?
This explores how a model can spend more or less thinking effort (tokens, samples, reasoning steps) depending on how hard each prompt is, instead of giving every question the same fixed budget.
This explores how an AI system can decide how much thinking to spend on each question, giving easy prompts a quick answer and saving the heavy compute for hard ones. The main finding in the corpus is simple. How you divide the compute matters about as much as how much compute you have. When the same total inference budget is shifted toward harder prompts, results improve enough that a smaller model with adaptive allocation can beat a larger model working under a uniform budget Can we allocate inference compute based on prompt difficulty?. Better budgeting can sometimes stand in for a bigger model.
The reason is that every problem has its own ideal amount of thinking. The token budget that gives the best accuracy changes with difficulty, so any fixed budget fails in two ways at once: it wastes tokens over-thinking easy problems and cuts hard problems off before the model reaches an answer Does thinking time need to scale with problem difficulty?. Over-thinking isn't only wasteful, either. Longer reasoning on a simple question can talk the model out of a correct first answer. Uniform budgets therefore cost accuracy as well as money.
You can also budget during training, not just at inference time. Models trained with token budgets that start generous and tighten gradually end up both more accurate and more efficient than models trained under one fixed limit Does gradually tightening token budgets beat fixed budget training?. The early generous phase lets the model discover strategies that work, and the later tight phase teaches it to compress them. It resembles how people learn: first you work a problem out at length, then you learn the shortcut.
There's also a connection most readers wouldn't expect: the prompt and the compute strategy shouldn't be designed separately. A prompt tuned without knowing whether the system will use best-of-N sampling or majority voting performs systematically worse. Optimizing the two together improves results by up to 50% Does prompt optimization without inference strategy fail?. The same 'budget by need' logic shows up in prompt compression too, where cutting more from demonstration examples and less from instructions and questions keeps accuracy while shrinking prompts up to 20x Can prompts be compressed without losing model performance?.
One gap is worth knowing about. These notes show that adaptive allocation works, but the material here says little about how a system estimates a prompt's difficulty before it has tried to solve it. That chicken-and-egg problem is where much of the practical engineering happens, and this set of notes doesn't cover it in depth.
Sources 5 notes
Research shows inference effectiveness varies dramatically by prompt difficulty. Reallocating the same total compute adaptively—giving easy prompts less and hard ones more—substantially outperforms larger models under uniform budgets.
Research shows the token budget that maximizes accuracy differs by problem difficulty. Fixed budgets either over-think easy problems or under-think hard ones, meaning adaptive allocation is a more efficient accuracy lever than increasing tokens uniformly.
Models trained with progressively tightening token budgets consistently achieve higher accuracy and better token efficiency than fixed-budget baselines. The approach works by separating learning into exploration (discovering strategies with generous budgets) and compression (distilling them under constraints).
Prompts optimized without knowledge of the inference strategy (best-of-N, majority voting) systematically underperform. Joint optimization of both prompt and inference strategy yields up to 50% improvement across reasoning and generation tasks.
LLMLingua achieves up to 20x prompt compression with only 1.5-point performance drops by allocating less compression to instructions and questions, more to demonstrations, then pruning tokens using a small model's perplexity scoring aligned to the target LLM.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Inference-Aware Prompt Optimization for Aligning Black-Box Large Language Models
- LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
- Does Thinking More always Help? Understanding Test-Time Scaling in Reasoning Models
- When More Thinking Hurts: Overthinking in LLM Test-Time Compute Scaling
- The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs
- Do LLMs Encode Functional Importance of Reasoning Tokens?
- Thinking as Compression: Your Reasoning Model is Secretly a Context Compressor
- Recursive Language Models