SYNTHESIS NOTE
Training, RL, and Test-Time Scaling Reasoning, Retrieval, and Evaluation Model Architecture and Internals
Do reasoning models switch between ideas too frequently?
Research explores whether o1-like models abandon promising reasoning paths prematurely by switching to different approaches without sufficient depth, and whether penalizing such transitions could improve accuracy.
"Thoughts Are All Over the Place" identifies a failure mode complementary to but distinct from overthinking: underthinking. Where overthinking generates excessively long traces, underthinking generates traces that switch between reasoning directions too frequently, failing to follow any promising path to completion.
The empirical finding: frequent thought switching correlates with incorrect responses across multiple o1-like models on challenging mathematical test sets. The model starts down one reasoning path, encounters difficulty, switches to a different approach, encounters difficulty there too, switches again — never committing enough depth to any single path to reach a solution.
A novel metric quantifies this: token efficiency in incorrect answers, measuring how much of the reasoning trace was "wasted" on abandoned approaches versus productively advancing toward a solution.
TIP (Thought-switching Penalty) is a pure decoding strategy — no model fine-tuning required. During generation, it penalizes the probability of tokens that signal thought transitions (linguistic markers like "Alternatively," "Let me try," "Wait"), encouraging the model to continue exploring the current path rather than jumping to a new one. The result: accuracy improves across challenging datasets.
This reframes the overthinking/underthinking relationship. They are not opposites on a single dimension (trace length). Overthinking is excessive computation within a committed path. Underthinking is insufficient computation per path due to premature switching. A model can simultaneously overthink (too many tokens total) and underthink (too few tokens per path) — producing a long trace that wanders between incomplete approaches.
The connection to Why do reasoning LLMs fail at deeper problem solving? is direct: premature thought switching is one mechanism that produces wandering behavior. The "unnecessary exploration" failure mode is exactly what happens when the model abandons productive branches for new ones without sufficient exploration.
Inquiring lines that use this note as a source 118
This note is a source for these synthesized inquiries. Follow a line forward into its question, or open it to trace back to all of its sources.
- Why do foundation models develop heuristics instead of world models?
- How do models integrate conflicting signals in reasoning tasks?
- When does knowledge activation fail across different model architectures?
- Can explicit constraint statements override the dominance of surface heuristics?
- Why do single examples trigger large reasoning improvements in models?
- Can penalizing reasoning transitions fix underthinking without fine-tuning models?
- Can reflection in reasoning models be corrective rather than just confirmatory?
- Can step-level deliberation flags guide other reasoning systems?
- Can a single SAE feature control reasoning behavior across model families?
- What makes bilevel metacognition architectural rather than emergent in current systems?
- Why do contrastive reasoning approaches outperform single-path belief evaluation?
- What role does exploration-exploitation balance play in abstraction formation?
- Does the model learn depth-wise drift as an explicit strategy?
- Does the reversal curse stem from the same one-way commitment architecture?
- Can multi-turn rewards fix models that lose track midway?
- What happens to chain-of-thought performance across distribution shifts?
- How does scene-switching prevent cross-problem interference in multi-agent reasoning?
- How should guidance levels adapt as the model's capability boundary shifts?
- Do reasoning models show the same answer-maintenance pattern that diffusion models exhibit?
- What makes intentional structure shifts different from segment boundaries?
- Why does fine-tuning degrade reasoning quality even as accuracy improves?
- What triggers overthinking versus underthinking in reasoning models?
- Can parallel thinking outperform sequential thinking under the same token budget?
- Does fine-tuning models for specific tasks destroy their ability to reason?
- How should iterative research tasks limit context per reasoning turn?
- How does reasoning instability prevent models from modeling individuals?
- Why does self-revision degrade reasoning accuracy in o1-like models?
- Why do longer reasoning chains signal hesitation rather than depth?
- Does reasoning structure match explicit versus implicit task demands?
- How does evaluation format change what we measure about model reasoning?
- How do foundation models develop task-specific heuristics instead of world models?
- Does reflection destabilize reasoning in dynamic environments?
- Why do reasoning models verbalize reasoning shortcuts less than necessary?
- What makes diverse reasoning sources more valuable than deeper single paths?
- Can inflection points in reasoning detect when models genuinely change their minds?
- Why does reflection in reasoning models stay confirmatory instead of corrective?
- Why does the same recalled information lead to different reasoning conclusions?
- Does thought consolidation address the confirmatory reflection problem in reasoning models?
- Why does inference-time thinking hurt proactive critical thinking in vanilla models?
- Why does intermediate step quality predict reasoning outcomes better than global features?
- How does RL refine reasoning paths without simply adding model capability?
- Why do reasoning chains degenerate into undirected exploration at scale?
- What happens to reasoning accuracy when models use more thinking tokens?
- Do base models and reasoning models fail in opposite directions on uncertainty?
- Can explicit optimal algorithms prevent reasoning model collapse at high complexity?
- How does graph of thoughts enable divide-and-conquer reasoning patterns?
- What makes multi-paradigm chaining a distinct reasoning topology?
- Why do reasoning models wander instead of searching systematically?
- Why do larger reasoning models show cyclicity only in later layers?
- What distinguishes redundant cycles from productive reconsidering cycles?
- Why does revision often make reasoning accuracy worse in frontier models?
- Why does reflection in reasoning models tend to be confirmatory rather than corrective?
- Why do long-horizon reasoning tasks need per-turn step limits rather than just compute budgets?
- Why do aha moments emerge specifically during the planning phase?
- Why does reasoning graph topology evolve differently across training phases?
- What distinguishes systematic search from wandering exploration in reasoning?
- Why do models follow a two-phase pattern of procedural then strategic learning?
- Can contrastive learning teach models to switch between logical and emotional reasoning?
- Which constraint types do reasoning models handle best?
- Why do longer reasoning chains correlate with lower accuracy in o1-like models?
- What changes when reasoning models adopt trajectory-response output formats?
- When are multiple independent attempts more valuable than depth?
- How does soft thinking compare to sampling multiple independent reasoning paths?
- Does this reasoning steering method work consistently across all model sizes?
- Why does reflection in reasoning models confirm rather than correct initial directions?
- Why do some reasoning steps receive negligible attention from later steps?
- Can static reasoning patterns work better than dynamic branch selection?
- How does collaboration itself become a degradation mechanism in reasoning tasks?
- Does internal self-revision actually degrade reasoning accuracy in models?
- Why does reasoning fine-tuning reduce a model's ability to abstain?
- Why do models skip steps that would make reasoning clearer?
- How does training data format shape which reasoning patterns emerge in models?
- Does unrestricted reasoning per search step degrade iterative quality over time?
- Do search agents face their own overthinking threshold like reasoning models do?
- What is the optimal balance between search rounds and reasoning depth per round?
- Does penalizing thought transitions improve reasoning without model retraining?
- Do reasoning models switch approaches when encountering local difficulty?
- How can prompt intervention reduce redundant reasoning steps dynamically?
- How much does switching overhead reduce reasoning token efficiency?
- Can models overthink and underthink at the same time?
- What mechanisms cause reasoning models to wander rather than focus?
- Why do per-turn thinking budgets matter alongside iterative retrieval depth?
- Why do some students restart entire projects instead of debugging incrementally?
- Why do different model training approaches produce different overthinking thresholds?
- Can layer-wise prediction stabilization identify when genuine reasoning has stopped?
- How do single wrong steps corrupt entire reasoning chains?
- What happens to model reasoning accuracy as thinking token requirements exceed critical thresholds?
- How does interaction horizon differ from chain-of-thought depth?
- How does making implicit reasoning requirements explicit change model performance?
- What failure modes emerge when scheme classification feeds downstream reasoning pipelines?
- What happens to iterative search quality when reasoning depth is unconstrained?
- How do progressive abstraction chains differ from branching reasoning topologies?
- Why do wrong numbers cost less accuracy than shuffled reasoning steps?
- Which tokens actually change across different reasoning paths in rollouts?
- What causes reasoning quality to degrade during long research tasks?
- Why does per-step deliberation lose global perspective compared to dynamic discovery?
- Does fine-tuning push models toward reasoning shortcuts that bypass the chain entirely?
- Why does vanilla GRPO cause mode collapse in hybrid reasoning settings?
- Can a single model implement fast thinking, slow thinking, and tool use?
- Why might chain-of-thought reasoning bypass action selection pathways?
- What makes answer equivalence sufficient to discard a reasoning path?
- Can metacognitive categories be learned instead of fixed by human designers?
- Why does reflection in reasoning models mostly confirm the first answer?
- Why do longer reasoning chains explore like tourists instead of scientists?
- What makes deterministic recursive reasoning models underperform on multi-solution tasks?
- Can auxiliary modules preserve reasoning without catastrophic forgetting?
- How does continuous soft thinking explore multiple paths without explicit training?
- What makes o1's chain-of-thought processing specifically effective for exploration tasks?
- How do past research mistakes prevent future pivot loops from repeating them?
- How should AI ideation systems decompose and recombine research concepts?
- Can we detect redundant reasoning steps during model inference instead of training?
- Why do reasoning traces fail to accurately reflect model decision-making?
- How do search and reasoning workflows improve forecasting performance over base models?
- Why does reflection in reasoning models often become theater rather than genuine thought?
- How does o1-style reasoning relate to learned search processes versus memorized solutions?
- What role do cyclic fixed points play in stable reasoning?
- What makes multi-turn critique trajectories more effective than single-turn reasoning chains?
- How does early commitment in reasoning differ from early exploitation in planning?
Related concepts in this collection 8
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Concept map
Do reasoning models switch between ideas too fre…
Why do reasoning LLMs fail at deeper problem solvi…
Does self-revision actually improve reasoning in l…
Does more thinking time always improve reasoning a…
Why do correct reasoning traces contain fewer toke…
Can minimal reasoning chains match full explanatio…
Does policy entropy collapse limit reasoning perfo…
Do iterative refinement methods suffer from overth…
Why do reasoning models overthink ill-posed questi…
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
your link
semantically near
linked from elsewhere
-
Why do reasoning LLMs fail at deeper problem solving?
Explores whether current reasoning models systematically search solution spaces or merely wander through them, and how this affects their ability to solve increasingly complex problems.
underthinking via switching is one mechanism producing the wandering pattern
-
Does self-revision actually improve reasoning in language models?
When o1-like models revise their own reasoning through tokens like 'Wait' or 'Alternatively', does this reflection catch and fix errors, or does it introduce new mistakes? This matters because self-revision is marketed as a key capability.
self-revision is a specific form of switching: the model revises (switches away from) an answer rather than deepening its current approach
-
Does more thinking time always improve reasoning accuracy?
Explores whether extending a model's thinking tokens linearly improves performance, or if there's a point beyond which additional reasoning becomes counterproductive.
the threshold may partly reflect switching overhead: tokens spent on transitions rather than productive reasoning
-
Why do correct reasoning traces contain fewer tokens?
In o1-like models, correct solutions are systematically shorter than incorrect ones for the same questions. This challenges assumptions that longer reasoning traces indicate better reasoning, and raises questions about what length actually signals.
incorrect traces are longer partly because switching generates wasted tokens
-
Can minimal reasoning chains match full explanations?
Does removing all explanatory text from chain-of-thought reasoning preserve accuracy? This tests whether verbose intermediate steps are necessary for solving problems or just artifacts of how language models are trained.
CoD addresses underthinking from the format side: minimal per-step drafts enforce depth within each step by eliminating the verbose intermediate context that enables thought-switching; where TIP penalizes switching tokens at decoding time, CoD prevents the runway for switching in the first place
-
Does policy entropy collapse limit reasoning performance in RL?
As reinforcement learning models become more confident in their policy choices, entropy drops and performance plateaus. Can we identify and counteract this bottleneck to sustain scaling?
training-time analog: entropy collapse reduces exploration diversity during training (narrowing the strategy repertoire), while underthinking reduces exploration depth during inference (abandoning strategies prematurely); both are exploration-exploitation failures at different timescales
-
Do iterative refinement methods suffer from overthinking?
Iterative refinement approaches like Self-Refine structurally resemble token-level overthinking in o1-like models. Does revision across multiple inference calls reproduce the same accuracy degradation seen within single inferences?
timescale generalization: underthinking operates within a single inference call (switching between reasoning threads); iterative refinement reproduces the same switching pattern across multiple inference calls — TIP-like penalties on transition tokens may apply at both timescales
-
Why do reasoning models overthink ill-posed questions?
Explores why models trained for extended reasoning produce drastically longer, less useful responses to unanswerable questions—and whether this represents a fixable training deficit or inherent limitation.
distinct but related failure: underthinking is premature switching between approaches (too shallow per path); overthinking on missing premises is inability to disengage (no valid path exists); both reveal the model lacks metacognitive control over its reasoning allocation
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs
- Meta-Reasoner: Dynamic Guidance for Optimized Inference-time Reasoning in Large Language Models
- Test-time Prompt Intervention
- Large Language Models Think Too Fast To Explore Effectively
- Fast, Slow, and Tool-augmented Thinking for LLMs: A Review
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
- LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!
- ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
Original note title
underthinking is premature thought switching — penalizing reasoning transitions improves accuracy without fine-tuning