Why does step-by-step reasoning degrade performance on judgment-based tasks?
This explores why forcing a model to 'think out loud' step-by-step can actually hurt on tasks that call for a judgment or direct read, rather than a multi-step derivation.
This explores why forcing a model to 'think out loud' can backfire on tasks that want a judgment rather than a derivation — and the corpus suggests the culprit isn't too little reasoning, but reasoning applied where it doesn't belong. The cleanest evidence comes from work showing that successful step-by-step prompting depends on the question's information flowing into the prompt *before* reasoning begins; for simple or judgment-style questions, a direct question-to-answer path beats a step-by-step one, because the optimal prompt shape depends on the question type, not the task label Why do some questions perform better without step-by-step reasoning?. In other words, chain-of-thought is a tool with a domain, and judgment tasks often fall outside it.
There's also a dosage problem. Accuracy follows an inverted-U against reasoning length — it peaks at intermediate chains and then declines, with the sweet spot getting *shorter* as models get more capable Why does chain of thought accuracy eventually decline with length?. Push past the threshold and benchmark accuracy can fall sharply (one study saw 87% collapse to 70% as thinking tokens ballooned), because models overthink easy problems and talk themselves out of correct snap judgments Does more thinking time always improve reasoning accuracy?. For a task where the right answer is closer to a first impression, every extra step is a chance to drift away from it.
The drift has a mechanism. Vanilla models often use 'thinking mode' to manufacture self-doubt that degrades performance — extended deliberation becomes second-guessing — and it takes RL training to redirect that same machinery from doubt into productive analysis Does extended thinking help or hurt model reasoning?. Relatedly, reasoning models wander down invalid paths and abandon promising ones prematurely, failing through disorganization rather than lack of compute Why do reasoning models abandon promising solution paths?. On judgment tasks, that wandering is pure downside: there's no long derivation to organize, just an answer to be talked out of.
The deeper reason cuts to what chain-of-thought actually is. It appears to be constrained imitation of reasoning *form*, not genuine inference — which is why logically *invalid* reasoning chains perform nearly as well as valid ones, since the model is matching the structure of reasoning, not its content Does logical validity actually drive chain-of-thought gains? Why does chain-of-thought reasoning fail in predictable ways?. When a task rewards a holistic call rather than a derivable chain, performing the *shape* of step-by-step thinking adds noise without adding signal. Tellingly, when researchers measured which steps the model actually attends to downstream, verification and backtracking steps drew almost no attention — you can prune ~75% of reasoning steps without losing accuracy Can reasoning steps be dynamically pruned without losing accuracy?. The steps that feel most like 'careful judgment' are often the most disposable.
The thing worth taking away: step-by-step reasoning isn't a universal upgrade you bolt onto any task. It's a procedure with a domain of validity, and on judgment tasks the model is essentially performing the costume of reasoning over an answer it could have read off directly — paying in drift, self-doubt, and wandering for a structure that doesn't fit the problem.
Sources 8 notes
Saliency analysis reveals that CoT prompting fails when question information doesn't aggregate into the prompt structure before reasoning begins. For simple questions, direct question-to-answer flow outperforms step-by-step reasoning, showing the optimal prompt depends on question type, not just task category.
Task accuracy peaks at intermediate CoT length, with optimal length increasing alongside task difficulty but decreasing with model capability. RL training naturally gravitates toward shorter chains as models improve, revealing that simplicity emerges from reward signals rather than explicit training.
Increasing thinking tokens from ~1,100 to ~16K reduced benchmark accuracy from 87.3% to 70.3%, revealing a non-monotonic relationship where models overthink easy problems and underthink hard ones.
Vanilla models use thinking mode counterproductively, inducing self-doubt that degrades performance. RL training reverses this, transforming the same mechanism into beneficial gap analysis. Training mediates reasoning quality, not just quantity.
Reasoning LLMs exhibit two reinforcing failures: wandering (invalid exploration) and underthinking (premature path-switching). Decoding-level interventions like thought-switching penalties improve accuracy without fine-tuning, suggesting viable solutions exist but are abandoned prematurely.
Illogical chain-of-thought exemplars matched valid CoT performance on BIG-Bench Hard, showing that structural properties—not logical validity—drive the gains. The model learns the form of reasoning, not genuine inference.
CoT guides models to pattern-match reasoning structure rather than perform genuine inference. This explains distribution-bounded failures, why structural coherence matters more than content correctness, and why performance optimizes against interpretability.
The PI framework categorizes reasoning into six types and uses attention maps to identify that verification and backtracking steps receive minimal downstream attention. Selecting only high-attention steps preserves accuracy while cutting reasoning length substantially.