Does more thinking always improve language model accuracy?
This explores whether longer reasoning chains and more 'thinking tokens' reliably make a model more accurate — and the corpus says no, the relationship is non-monotonic and quality matters more than quantity.
This explores whether more thinking always helps a language model get the right answer. The short version from the collection: it doesn't, and the more interesting finding is *why*. Accuracy tends to follow an inverted-U — it climbs with a bit of reasoning, peaks, then declines. One study found that pushing thinking tokens from around 1,100 up to 16,000 actually dropped benchmark accuracy from 87% to 70%, because models overthink easy problems and (oddly) underthink hard ones Does more thinking time always improve reasoning accuracy?. A separate line of work formalizes that curve: the *optimal* chain-of-thought length rises with task difficulty but falls as the model gets more capable, so stronger models naturally prefer shorter chains Why does chain of thought accuracy eventually decline with length?.
The deeper point is that 'more thinking' isn't one thing — it can be productive analysis or it can be self-sabotage. One paper shows that vanilla models often use extended thinking to second-guess themselves, inducing self-doubt that *degrades* performance; reinforcement-learning training reverses this, redirecting the same mechanism into useful gap-analysis. Training changes the quality of reasoning, not just the amount Does extended thinking help or hurt model reasoning?. So the right question isn't 'how long should it think' but 'is the thinking any good.'
That reframes the goal as knowing *when* to think at all. One approach trains a single model to route between extended reasoning and direct, concise answers — picking the mode per problem without explicit difficulty labels — so it doesn't waste a long chain on a question it could answer instantly Can models learn when to think versus respond quickly?. This matters because not every token in a reasoning trace is pulling weight: only about 20% are high-entropy 'forking' decision points that actually drive learning and outcomes, and the rest are largely filler Do high-entropy tokens drive reasoning model improvements?. Some models even compute the correct answer in their first few layers and then overwrite it with format-compliant filler tokens — the 'thinking' you see on the page isn't where the reasoning happened Do transformers hide reasoning before producing filler tokens?.
There are also hard ceilings that no amount of extra thinking can push through. Reasoning accuracy drops sharply just from longer *inputs* — falling from 92% to 68% with only 3,000 tokens of padding, well below the context limit, and chain-of-thought doesn't rescue it Does reasoning ability actually degrade with longer inputs?. And much of what looks like reasoning error is actually local memorization leaking in from preceding tokens, accounting for up to two-thirds of mistakes as problems get harder Where do memorization errors arise in chain-of-thought reasoning?. More fundamentally, a model can't think its way past its own verification gap: self-improvement is formally bounded, and every reliable correction needs something external to validate it What stops large language models from improving themselves?.
The thing you might not have expected to learn: the most promising route isn't longer thinking but *better-shaped* thinking. Training models on messy search processes — including dead ends and backtracking, not just clean optimal solutions — produces 25% stronger problem-solvers, because they learn to explore and adapt rather than recite a fixed path Does training on messy search processes improve reasoning?. Quantity is a red herring; structure, calibration, and knowing when to stop are the real levers.
Sources 10 notes
Increasing thinking tokens from ~1,100 to ~16K reduced benchmark accuracy from 87.3% to 70.3%, revealing a non-monotonic relationship where models overthink easy problems and underthink hard ones.
Task accuracy peaks at intermediate CoT length, with optimal length increasing alongside task difficulty but decreasing with model capability. RL training naturally gravitates toward shorter chains as models improve, revealing that simplicity emerges from reward signals rather than explicit training.
Vanilla models use thinking mode counterproductively, inducing self-doubt that degrades performance. RL training reverses this, transforming the same mechanism into beneficial gap analysis. Training mediates reasoning quality, not just quantity.
Thinkless trains a single model to select between extended reasoning and direct responses using DeGRPO, which decouples mode selection from answer refinement. This prevents mode collapse and enables self-calibrated routing without explicit difficulty labels.
Only ~20% of tokens exhibit high entropy as pivotal reasoning decision points; RLVR primarily adjusts these forking tokens. Training exclusively on them matches or exceeds full-gradient performance, revealing that the minority carries the learning signal.
Logit lens analysis shows models trained with hidden CoT tokens compute correct answers in layers 1-3, then actively suppress these representations in final layers to produce format-compliant filler output. The reasoning is fully recoverable from lower-ranked token predictions.
FLenQA shows reasoning accuracy drops from 92% to 68% at just 3000 tokens of padding, far below context window capacity. The degradation is task-agnostic, uncorrelated with language modeling performance, and persists even with chain-of-thought prompting.
STIM framework identifies local, mid-range, and long-range memorization sources in CoT reasoning. Local memorization—based on preceding tokens—accounts for up to 67% of reasoning errors, especially as complexity increases and distributional shift occurs.
Self-improvement in LLMs is formally bounded by the generation-verification gap, meaning every reliable fix requires something external to validate and enforce it. Models cannot escape this constraint through metacognition alone.
Stream of Search pretraining, which represents exploration and backtracking as serialized strings, achieves 25% higher accuracy than optimal-trajectory-only training. Models learn internal world models for search and adaptive strategies rather than fixed external methods.