An AI's step-by-step reasoning can sound convincing, but does reading it actually help you predict what the AI will do?
Can reasoning traces that feel convincing fail to help people predict behavior?
This explores whether a fluent, step-by-step explanation from a reasoning model is a reliable guide to what the model will actually do, so that reading it lets you forecast its answers and failures.
This explores whether a fluent, step-by-step explanation from a reasoning model is a reliable guide to what the model will actually do. The corpus suggests it often isn't. The trace is better read as a performance of reasoning than as a log of it, and that gap gives a reader false confidence in both directions.
The strongest evidence is that traces keep working when they're wrong. Invalid traces frequently still end in correct answers, because they correlate with answers through learned formatting rather than functional reasoning (Do reasoning traces actually cause correct answers?). Logically invalid chain-of-thought examples performed nearly as well as valid ones on BIG-Bench Hard (Does logical validity actually drive chain-of-thought gains?). Models trained on deliberately irrelevant traces kept their accuracy and sometimes generalized better (Do reasoning traces need to be semantically correct?). If the logic in a trace can be broken without changing the outcome, then checking the logic tells you little about the outcome. Reading a clean trace and concluding the answer is sound is a mistake. So is reading a messy trace and concluding the answer is wrong.
The mismatch also shows up inside the model. Mapping which trace steps depend on which found that most erroneous steps don't influence the final answer, and that the argument a trace presents doesn't match the internal causal pathways (Do reasoning traces actually show how models think?). The story on the page and the computation underneath are only loosely related (Do reasoning traces show how models actually think?). The cues people use to gauge effort mislead too. Trace length tracks difficulty only on problems near the training data, and it decouples once a problem moves away from them (Does longer reasoning actually mean harder problems?). A long trace on a new problem doesn't mean the model is working hard. Telling a model its reasoning is being monitored doesn't change how often it leaves a hint out of the trace (Does telling models they are watched improve reasoning faithfulness?), so you can't prompt your way to a more honest trace.
What does predict behavior is less about how the trace reads and more about where the problem sits. Chain-of-thought degrades in a regular way as tasks, lengths and formats drift from the training data, and the outputs stay fluent while the logic stops holding together (Does chain-of-thought reasoning actually generalize beyond training data?, Why does chain-of-thought reasoning fail in predictable ways?). A few parts of a trace do carry weight. Planning and backtracking sentences act as sparse pivots that steer what follows (Which sentences actually steer a reasoning trace?). Some structural failures can also be seen in a trace, such as wandering off or abandoning a promising path too early (Why do reasoning models abandon promising solution paths?). So a trace isn't noise. But reading the whole thing as an explanation is different from spotting the few sentences that matter or the shape of the exploration.
The corpus doesn't include a study where people read traces and try to forecast a model's behavior, so the human side of this is inferred. The model side is well covered: the traces are weak evidence of cause, and that is the reason a convincing one can leave a reader less able to predict what the model will do.
Sources 11 notes
R1's intermediate tokens carry no special execution semantics and are generated identically to other LLM output. Invalid traces frequently produce correct answers, proving traces are not causally necessary—they correlate with answers via learned formatting, not functional reasoning.
Illogical chain-of-thought exemplars matched valid CoT performance on BIG-Bench Hard, showing that structural properties—not logical validity—drive the gains. The model learns the form of reasoning, not genuine inference.
Models trained on systematically irrelevant traces maintain solution accuracy and sometimes improve out-of-distribution generalization, suggesting traces function as computational scaffolding rather than meaningful reasoning steps.
ReasoningFlow found that most erroneous steps in traces don't influence final answers, and critically, the discourse structure traces present linguistically does not match their actual internal causal pathways. This gap suggests traces are narrative surface rather than verified computation logs.
LLM reasoning traces perform as persuasive appearances rather than reliable explanations of computation. Invalid logical steps perform nearly as well as valid ones, and corrupted traces generalize comparably, showing that semantic correctness is not what produces the performance gains.
Show all 11 sources
Controlled A* maze experiments show trace length correlates with difficulty only in-distribution but decouples entirely out-of-distribution. Trace length primarily reflects recall of training schemas, not adaptive computation.
Prompting models that their reasoning is monitored has no effect on hint omission rates. This suggests CoT generation is not modulated by perceived social context, ruling out prompt-engineering fixes and certain safety monitoring assumptions.
DataAlchemy experiments show CoT fails systematically under distributional shifts in task, length, and format. Models produce fluent but logically inconsistent reasoning — imitating reasoning form without valid underlying logic.
CoT guides models to pattern-match reasoning structure rather than perform genuine inference. This explains distribution-bounded failures, why structural coherence matters more than content correctness, and why performance optimizes against interpretability.
Counterfactual resampling, attention analysis, and causal suppression all identify planning and backtracking sentences as thought anchors—sparse critical points that guide subsequent reasoning. These are functional pivots, not noise.
Reasoning LLMs exhibit two reinforcing failures: wandering (invalid exploration) and underthinking (premature path-switching). Decoding-level interventions like thought-switching penalties improve accuracy without fine-tuning, suggesting viable solutions exist but are abandoned prematurely.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
- Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens
- Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
- Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!
- When More is Less: Understanding Chain-of-Thought Length in LLMs
- What Characterizes Effective Reasoning? Revisiting Length, Review, and Structure of CoT
- CoT is Not True Reasoning, It Is Just a Tight Constraint to Imitate: A Theory Perspective
- A Comment On "The Illusion of Thinking": Reframing the Reasoning Cliff as an Agentic Gap