Why do corrupted reasoning traces sometimes generalize better than correct ones?
This explores a surprising finding in the corpus: that reasoning traces seem to work as computational scaffolding rather than meaningful logic, so even deliberately broken ones can teach a model — and sometimes transfer better to new problems than clean ones.
This explores why corrupted reasoning traces sometimes generalize better than correct ones — and the corpus's answer is unsettling: it's because the trace was probably never doing the reasoning in the first place. When models are trained on systematically irrelevant or scrambled traces, they keep their solution accuracy and occasionally improve out-of-distribution generalization, which suggests the trace functions as computational scaffolding — a structured run-up that gives the model more steps to compute — rather than a chain of meaningful logical moves Do reasoning traces need to be semantically correct?. If the content of the steps isn't what's carrying the answer, corrupting that content costs little.
A cluster of notes converges on the same root cause: the trace is stylistic mimicry, not verified causal reasoning. The intermediate tokens in models like R1 are generated identically to any other output and carry no special execution semantics, and invalid traces routinely produce correct answers — so traces correlate with answers through learned formatting, not function Do reasoning traces actually cause correct answers?. Chain-of-thought turns out to be constrained imitation: format effects dominate content, and structurally invalid prompts still succeed What makes chain-of-thought reasoning actually work?. Two independent notes state the punchline almost verbatim — corrupted traces generalize comparably, so semantic correctness is not what produces the performance gain Do reasoning traces show how models actually think?.
Here's the part you might not expect: corruption can actively help generalization. Clean traces tend to encode the schema of the training distribution. Trace length, for instance, tracks how close a problem sits to training data rather than how hard it is — it reflects recalled schemas, not adaptive thinking Does longer reasoning actually mean harder problems?. And the single largest source of reasoning errors is local memorization, where the model parrots patterns from immediately preceding tokens, accounting for up to two-thirds of mistakes as problems drift out of distribution Where do memorization errors arise in chain-of-thought reasoning?. A correct trace gives the model a clean, memorizable path to overfit to. A corrupted one denies it that crutch — so when the model can't lean on the surface pattern, what's left is the scaffolding effect that transfers more cleanly to unfamiliar problems.
The doorway worth walking through is what this does NOT mean: it doesn't mean traces are useless. The same corpus shows that WHICH parts of a trace matter is highly uneven. Planning and backtracking sentences act as sparse 'thought anchors' that genuinely steer what follows Which sentences actually steer a reasoning trace?, and verifying the process step-by-step rather than scoring final answers raised task success from 32% to 87% Where do reasoning agents actually fail during long traces?. Step-level confidence filtering beats global averaging precisely because local quality is where breakdowns hide Does step-level confidence outperform global averaging for trace filtering?. So the corruption result isn't 'reasoning doesn't exist' — it's that bulk filler content is interchangeable while a few structural pivots are load-bearing.
The deeper takeaway is that CoT's gains are distribution-bounded: under shifts in task, length, or format, models produce fluent but logically inconsistent reasoning, imitating the form without the logic Does chain-of-thought reasoning actually generalize beyond training data?, and frontier reasoning models collapse to ~20-23% on constraint-satisfaction problems demanding real backtracking Can reasoning models actually sustain long-chain reflection?. Corrupted traces generalizing well is the flip side of correct traces generalizing poorly — both are symptoms of the same fact: the model is matching the shape of reasoning, and the shape, it turns out, survives corruption.
Sources 11 notes
Models trained on systematically irrelevant traces maintain solution accuracy and sometimes improve out-of-distribution generalization, suggesting traces function as computational scaffolding rather than meaningful reasoning steps.
R1's intermediate tokens carry no special execution semantics and are generated identically to other LLM output. Invalid traces frequently produce correct answers, proving traces are not causally necessary—they correlate with answers via learned formatting, not functional reasoning.
CoT systems reproduce the form of reasoning through pattern matching rather than performing genuine logical inference. This explains why format effects dominate content, why structurally invalid prompts succeed, and why stronger reasoning models become less instruction-compliant.
LLM reasoning traces perform as persuasive appearances rather than reliable explanations of computation. Invalid logical steps perform nearly as well as valid ones, and corrupted traces generalize comparably, showing that semantic correctness is not what produces the performance gains.
Controlled A* maze experiments show trace length correlates with difficulty only in-distribution but decouples entirely out-of-distribution. Trace length primarily reflects recall of training schemas, not adaptive computation.
STIM framework identifies local, mid-range, and long-range memorization sources in CoT reasoning. Local memorization—based on preceding tokens—accounts for up to 67% of reasoning errors, especially as complexity increases and distributional shift occurs.
Counterfactual resampling, attention analysis, and causal suppression all identify planning and backtracking sentences as thought anchors—sparse critical points that guide subsequent reasoning. These are functional pivots, not noise.
Reliability for long-trace reasoning comes from checking intermediate states and policy compliance during generation, not from scoring final outputs. Adding intermediate verification raised task success from 32% to 87% because most failures are process violations, not wrong answers.
Local step-level confidence catches reasoning breakdowns that global averaging masks and enables early stopping before traces complete. This approach achieves comparable accuracy gains to naive majority voting with far fewer generated traces, proving trace quality matters more than quantity.
DataAlchemy experiments show CoT fails systematically under distributional shifts in task, length, and format. Models produce fluent but logically inconsistent reasoning — imitating reasoning form without valid underlying logic.
DeepSeek-R1 and o1-preview achieve only 20-23.6% exact match on 850 constraint satisfaction problems requiring genuine backtracking. This ceiling reveals that reflective reasoning fluency does not translate to actual problem-solving competence on unfamiliar instance structures.