How should inference budgets adapt based on prompt difficulty?
This explores how a model can spend more or less inference compute depending on how hard a given prompt is — and what the corpus says actually controls whether that extra compute pays off.
This explores how inference budgets should flex with prompt difficulty — spending little on easy prompts and more on hard ones — and, more interestingly, what determines whether the extra spend is even worth it. The starting point is the clearest result in the corpus: under a fixed total compute pool, reallocating it adaptively (less for easy prompts, more for hard ones) beats giving every prompt the same budget — and can outperform simply using a bigger model under a uniform budget Can we allocate inference compute based on prompt difficulty?. So the question isn't whether to adapt, but how to decide difficulty and route compute accordingly.
The hard part is the routing decision, and the corpus suggests a model can learn it for itself rather than relying on hand-labeled difficulty. One approach trains a single model to choose between extended 'thinking' and a quick direct answer, decoupling the *mode choice* from *answer quality* so the model doesn't collapse into always-think or always-skip — a self-calibrated router with no explicit difficulty labels Can models learn when to think versus respond quickly?. A useful signal for that router is the model's own confidence: high-confidence outputs are stable and barely move when you perturb the prompt, while low confidence produces big swings Does model confidence predict robustness to prompt changes? — exactly the prompts where more compute might help.
But a sharp twist comes from work on *why* models fail at reasoning. Failures aren't driven by task complexity hitting some threshold; they're driven by instance-level *unfamiliarity* — the model breaks on prompts unlike anything it was trained on, not on 'long' or 'complex' ones Do language models fail at reasoning due to complexity or novelty?. That reframes 'difficulty': the prompts that most need extra budget are the *novel* ones, not the superficially complicated ones — and throwing more tokens at a genuinely unfamiliar instance may not rescue it at all.
This sets up the budget's ceiling. More inference compute doesn't make a non-reasoning model catch up to a reasoning model, because the gap is about a trained reasoning protocol that makes extra tokens *productive*, not about raw token count Can non-reasoning models catch up with more compute?. And not all tokens are equal: roughly 20% of tokens are high-entropy 'forking points' that actually steer reasoning Do high-entropy tokens drive reasoning model improvements?. Adaptive budgeting, then, is less about *quantity* of compute and more about spending it at the decision points that matter — and only when the model has been trained to use it.
One more wrinkle worth knowing: the budget shouldn't be tuned in isolation from the prompt. Prompts optimized without knowledge of the inference strategy (best-of-N, majority voting) systematically underperform, and jointly optimizing prompt and inference strategy can yield up to 50% improvement Does prompt optimization without inference strategy fail?. Even step-by-step reasoning itself is sometimes the wrong spend — for simple questions, direct question-to-answer flow beats forcing a reasoning chain Why do some questions perform better without step-by-step reasoning?. The throughline: adaptive inference budgeting works best as a joint decision over difficulty, prompt shape, and inference mode — not a single dial labeled 'more compute.'
Sources 8 notes
Research shows inference effectiveness varies dramatically by prompt difficulty. Reallocating the same total compute adaptively—giving easy prompts less and hard ones more—substantially outperforms larger models under uniform budgets.
Thinkless trains a single model to select between extended reasoning and direct responses using DeGRPO, which decouples mode selection from answer refinement. This prevents mode collapse and enables self-calibrated routing without explicit difficulty labels.
ProSA found that when models are highly confident, they resist prompt rephrasing; low confidence causes major output swings. Larger models, few-shot examples, and objective tasks all correlate with higher confidence and greater robustness.
LRMs don't break at complexity thresholds but at instance-novelty boundaries. Models fit instance-based patterns rather than generalizable algorithms, so any reasoning chain succeeds if trained on similar instances, regardless of length.
Reasoning models persistently outperform non-reasoning models regardless of inference budget because training instills a reasoning protocol that makes additional tokens productive. The gap is fundamentally about deployment mechanisms and training structure, not raw capability.
Only ~20% of tokens exhibit high entropy as pivotal reasoning decision points; RLVR primarily adjusts these forking tokens. Training exclusively on them matches or exceeds full-gradient performance, revealing that the minority carries the learning signal.
Prompts optimized without knowledge of the inference strategy (best-of-N, majority voting) systematically underperform. Joint optimization of both prompt and inference strategy yields up to 50% improvement across reasoning and generation tasks.
Saliency analysis reveals that CoT prompting fails when question information doesn't aggregate into the prompt structure before reasoning begins. For simple questions, direct question-to-answer flow outperforms step-by-step reasoning, showing the optimal prompt depends on question type, not just task category.