Why do longer reasoning chains explore like tourists instead of scientists?
This explores why models that 'think longer' often wander and abandon promising paths rather than systematically searching — and what the corpus says is actually going wrong inside long reasoning chains.
This explores why longer reasoning chains tend to wander — sampling a bit of everything and bailing early — instead of committing to a path and testing it the way a scientist would. The corpus traces the problem to two reinforcing failures named directly in Why do reasoning models abandon promising solution paths?: *wandering* (exploring invalid territory) and *underthinking* (switching away from a good path before it pays off). The striking part is that the fix doesn't require more compute or retraining — a simple decoding-level penalty on thought-switching tokens recovers accuracy, which means the better solution was already in reach and the model just walked away from it. Do reasoning models switch between ideas too frequently? confirms this independently: o1-style models abandon paths mid-exploration, and penalizing the transition tokens alone improves hard-math accuracy with no fine-tuning.
So if longer isn't better, why does the field keep scaling length? Part of the answer is that length was never a reliable signal of real thinking. Does longer reasoning actually mean harder problems? shows trace length tracks how close a problem sits to the training distribution, not how hard it actually is — long chains are often recall of familiar schemas dressed up as deliberation. Why does chain of thought accuracy eventually decline with length? sharpens this into a curve: accuracy peaks at an intermediate length and *declines* past it, and more capable models naturally prefer shorter chains. The tourist, in other words, keeps walking long after the scientist would have stopped.
The deeper diagnosis is structural, not quantitative. Why do large language models explore less effectively than humans? offers a mechanistic reason for the touristy behavior: uncertainty signals dominate the *early* transformer layers while the 'empowerment' signals that reward long-term exploration only emerge in the *middle* layers — so the model commits to a direction before the part of it that values deep exploration can weigh in. Reasoning-trained models partly overcome this by buying time. And Does extended thinking help or hurt model reasoning? shows the raw 'think more' mechanism is double-edged: untrained, extended thinking breeds self-doubt and path-abandonment; RL redirects the same mechanism into productive gap analysis. Quality of thinking, not quantity, is what training actually changes.
The most useful contrast in the corpus is what scientific exploration would look like instead. Can abstractions guide exploration better than depth alone? (RLAD) argues that the cure for depth-only wandering is *breadth-first structure*: spending test-time compute on a few diverse high-level abstractions, then pursuing them, beats sampling many parallel solution attempts — exactly the structured search a scientist runs. Two adjacent findings explain why brute-force length can't substitute for this. Why do reasoning models overthink ill-posed questions? shows models trained only to *produce* steps never learn when to *disengage*, so they pour reasoning into ill-posed questions a non-reasoning model rejects instantly. And Do language models fail at reasoning due to complexity or novelty? shows models fit instance-level patterns rather than general algorithms — so a longer chain on an unfamiliar instance is just more wandering, not more figuring-out. The thing you didn't know you wanted to know: the cheapest gains here come not from thinking longer but from knowing when to stop switching, when to go broad before going deep, and when to refuse the question entirely.
Sources 9 notes
Reasoning LLMs exhibit two reinforcing failures: wandering (invalid exploration) and underthinking (premature path-switching). Decoding-level interventions like thought-switching penalties improve accuracy without fine-tuning, suggesting viable solutions exist but are abandoned prematurely.
o1-like models frequently abandon reasoning paths mid-exploration, wasting tokens on incomplete approaches. A decoding-only penalty on thought-transition tokens (TIP strategy) discourages switching, improving accuracy on challenging math without model fine-tuning.
Controlled A* maze experiments show trace length correlates with difficulty only in-distribution but decouples entirely out-of-distribution. Trace length primarily reflects recall of training schemas, not adaptive computation.
Task accuracy peaks at intermediate CoT length, with optimal length increasing alongside task difficulty but decreasing with model capability. RL training naturally gravitates toward shorter chains as models improve, revealing that simplicity emerges from reward signals rather than explicit training.
SAE decomposition shows uncertainty values dominate early transformer blocks while empowerment representations emerge only in middle blocks. This temporal mismatch causes models to commit to decisions before long-term exploration signals can influence them. Reasoning-trained o1 overcomes this by extending computation time.
Vanilla models use thinking mode counterproductively, inducing self-doubt that degrades performance. RL training reverses this, transforming the same mechanism into beneficial gap analysis. Training mediates reasoning quality, not just quantity.
RLAD jointly trains abstraction and solution generators, showing that allocating test-time compute to diverse abstractions outperforms parallel solution sampling at large budgets. Abstractions create structured breadth-first exploration that prevents the underthinking failure mode of depth-only reasoning chains.
Reasoning models generate redundant, lengthy responses to questions with missing premises while non-reasoning models correctly identify them as unanswerable. Training optimizes for producing reasoning steps but never teaches models when to disengage.
LRMs don't break at complexity thresholds but at instance-novelty boundaries. Models fit instance-based patterns rather than generalizable algorithms, so any reasoning chain succeeds if trained on similar instances, regardless of length.