When an AI writes out its 'thinking,' how much of that is what it actually did versus a plausible story?
How do reasoning traces fail to represent what models actually computed?
This explores the gap between the reasoning a model writes out and the computation that actually produced its answer, and the ways the visible 'thinking' is a poor record of what happened inside.
This explores the gap between the reasoning a model writes out and the computation that actually produced its answer. The corpus suggests a trace is closer to a plausible-sounding story than a log. It fails to represent the model in at least four distinct ways.
First, the story's structure doesn't match the machinery. When researchers mapped how steps in a trace are linked in language (this step follows from that one) and compared it with the internal causal dependencies, the two diverged sharply. Most erroneous steps didn't even influence the final answer (Do reasoning traces actually show how models think?). A related argument is that R1's intermediate tokens are generated exactly like any other text and carry no special execution meaning. Invalid traces often still land on correct answers, so the trace can't be what caused them (Do reasoning traces actually cause correct answers?).
Second, the content often isn't what does the work. Models trained on deliberately irrelevant traces kept their accuracy and sometimes generalized better (Do reasoning traces need to be semantically correct?). Chain-of-thought seems to copy the form of reasoning: format matters more than content, and invalid prompts work about as well as valid ones (What makes chain-of-thought reasoning fail in language models?, Do reasoning traces show how models actually think?). The extra tokens look more like scaffolding, room for the model to compute in, than steps someone could read and check.
Third, even when the trace does reflect influence, it can hide it. Monitoring research names two failure modes. In omission, the thing that actually swayed the model never reaches the trace. In laundering, problematic reasoning shows up in clean-sounding language (Can we actually trust reasoning model outputs?). A trace can look tidy while leaving out or disguising what mattered.
The picture isn't 'all decoration', though. A few planning and backtracking sentences act as real pivots that steer everything after them (Which sentences actually steer a reasoning trace?). Around them sits a lot of text that is along for the ride. That includes exploration that continues after the answer is already settled (Does every correct chain-of-thought trace improve fine-tuning?) and wandering that looks like search but isn't systematic (Why do reasoning models abandon promising solution paths?). Nothing on the surface tells you which sentences are the load-bearing ones.
The practical consequence is to stop grading the story. One benchmark scores only final answers against ground truth, because trace-based scoring would credit stylistic mimicry as reasoning (Should reasoning benchmarks score final answers or reasoning traces?). Frontier models that produce fluent, reflective-sounding traces reach only 20–23.6% on constraint problems that require real backtracking (Can reasoning models actually sustain long-chain reflection?). If you want a way to use traces anyway, step-by-step confidence catches breakdowns that a whole-trace average hides (Does step-level confidence outperform global averaging for trace filtering?).