How do thinking tokens exhibit diminishing returns beyond a critical threshold?
This explores why making a model 'think' longer stops helping past a certain point — and what the corpus says is actually going on when extra reasoning tokens start to hurt rather than help.
This explores why making a model 'think' longer stops helping past a certain point. The cleanest evidence is also the most striking: pushing thinking tokens from around 1,100 up to 16,000 dropped benchmark accuracy from 87.3% to 70.3% — not a plateau but an actual decline Does more thinking time always improve reasoning accuracy?. The relationship is non-monotonic: models overthink easy problems (burning tokens past the point of usefulness) and underthink hard ones. So 'diminishing returns' undersells it — beyond the threshold, more thinking can be net negative.
Why does the extra thinking go bad rather than just flat? One clue is that the useful signal in a reasoning trace is concentrated, not spread evenly. Specific tokens — 'Wait,' 'Therefore' — spike in mutual information with the correct answer, while most tokens carry little; suppress the peaks and accuracy falls, suppress random tokens and nothing happens Do reflection tokens carry more information about correct answers?. If the value lives in a few inflection points, then extending the chain mostly adds low-information filler that dilutes and destabilizes. A sharper version of the same suspicion: reasoning traces may be learned formatting rather than functional computation — invalid traces routinely produce correct answers — so generating more of them isn't adding more 'reasoning,' just more stylistic mimicry Do reasoning traces actually cause correct answers?.
The threshold itself is frustratingly invisible. It shifts with model, task, and difficulty, and there's no reliable predictor — you only know you crossed it after accuracy starts dropping, though difficulty estimators and runtime confidence signals can sometimes catch it dynamically How can we predict the optimal thinking token threshold?. This is also not unique to chain-of-thought: deep research agents that take more search steps follow the same test-time scaling curve, with the same diminishing returns, suggesting this is a general property of spending more inference compute rather than a quirk of one method Do search steps follow the same scaling rules as reasoning tokens?.
The more interesting turn is what the corpus says actually beats 'think longer.' Spending the same token budget on several independent shorter chains and majority-voting outperforms extending one chain by up to 22% — parallel diversity samples the model's capability more faithfully, while sequential extension mostly inflates variance without improving correctness Why does parallel reasoning outperform single chain thinking?. And on the training side, curricula that start with a generous token budget and gradually tighten it beat fixed budgets: let the model explore strategies with room, then compress them under constraint Does gradually tightening token budgets beat fixed budget training?. Both point the same way — the gains come from how reasoning is structured and allocated, not from raw length.
Worth knowing: if verbalized tokens are partly a training artifact, the whole length axis may be the wrong knob. Latent-reasoning architectures (Coconut, Heima, depth-recurrent models) scale test-time compute through hidden-state iteration with no visible tokens at all Can models reason without generating visible thinking tokens?, hinting that 'how many thinking tokens' is a question we may eventually stop asking.
Sources 8 notes
Increasing thinking tokens from ~1,100 to ~16K reduced benchmark accuracy from 87.3% to 70.3%, revealing a non-monotonic relationship where models overthink easy problems and underthink hard ones.
Specific tokens like "Wait" and "Therefore" show sharp spikes in mutual information with correct answers. Suppressing them harms reasoning while suppressing equal random tokens does not, and representation recycling improves accuracy 20%.
R1's intermediate tokens carry no special execution semantics and are generated identically to other LLM output. Invalid traces frequently produce correct answers, proving traces are not causally necessary—they correlate with answers via learned formatting, not functional reasoning.
The overthinking threshold depends on task difficulty, model training, and domain, but remains invisible until crossed. Recent work suggests difficulty estimators and runtime confidence signals can detect thresholds dynamically.
Deep research agents improve with more search steps in a pattern mirroring the reasoning-token relationship, with both exhibiting diminishing returns. This reveals a new inference-compute axis beyond model capability alone.
Multiple independent reasoning paths with majority voting achieve up to 22% higher accuracy than extending a single chain under the same token budget. Parallel diversity samples reasoning capability more faithfully than sequential extension, which inflates variance without improving correctness.
Models trained with progressively tightening token budgets consistently achieve higher accuracy and better token efficiency than fixed-budget baselines. The approach works by separating learning into exploration (discovering strategies with generous budgets) and compression (distilling them under constraints).
Multiple architectures—depth-recurrent models, Heima, and Coconut—demonstrate that test-time compute scales through hidden state iteration rather than token generation. This suggests verbalization is a training artifact, not a reasoning requirement.