SYNTHESIS NOTE
Topics›Reasoning o1 o3 Search›this note

Do reasoning traces need to be semantically correct?

Can models learn to solve problems from deliberately corrupted or irrelevant reasoning traces? This challenges assumptions about what makes intermediate tokens useful for learning.

Synthesis note · 2026-02-22 · sourced from Reasoning o1 o3 Search
How should we spend compute at inference time? What kind of thing is an LLM really?

"Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens" presents the strongest evidence yet against the assumption that reasoning traces carry meaningful semantics that contribute to solution quality.

The experimental design is clean. Transformers are trained on A* search traces for shortest-path planning in random mazes. Three conditions: (1) correct traces, (2) no traces, and (3) deliberately corrupted traces that have no relation to the specific problem they are paired with. The corrupted traces are not just noisy — they are systematically irrelevant, paired with wrong problems.

The results: corrupted-trace models maintain performance largely consistent with correct-trace models. In some cases they improve on correct-trace models and generalize more robustly to out-of-distribution tasks. Models trained on entirely correct traces still produce invalid reasoning traces when arriving at correct solutions — the formal A* validator confirms only a loose correlation between trace accuracy and solution accuracy.

This result directly challenges three assumptions simultaneously. First, that intermediate tokens function as reasoning steps (they may function as computational scaffolding — additional forward passes — regardless of semantic content). Second, that correct traces are superior training data (the scaffolding hypothesis predicts that any tokens providing additional computation would work). Third, that the "aha moment" in DeepSeek R1 indicates genuine realization (a single token insertion does not change internal state; it provides one more forward pass).

The "Stop Anthropomorphizing" position paper reinforces this from a different angle. It argues the community's tendency to call intermediate tokens "thoughts" or "reasoning traces" is actively harmful — generating false confidence and directing research toward improving trace quality rather than understanding the computational mechanism. The LLM-Modulo framework (generate-test with external verification) is proposed as the principled alternative: treat the LLM as a generator, use sound external verifiers for guarantees.

The practical implication: optimizing trace "interpretability" or "correctness" may be orthogonal to optimizing solution accuracy. The traces most useful for model performance may be those that provide optimal computational scaffolding, not those that most closely resemble human reasoning. This converges with What do models actually learn from chain-of-thought training?, which shows from the opposite direction that structural perturbations (shuffled steps) cause severe accuracy drops while content perturbations (wrong numbers, removed keywords) cause minimal impact. Together, these findings isolate the active ingredient: logical architecture, not semantic content.

Theoretical backing (RL-STaR): The theoretical analysis of the STaR framework provides formal support: RL-based self-taught reasoning can improve capabilities despite incorrect reasoning steps in the training data, because the iterative policy gradient converges under bounded error conditions. The model doesn't need correct intermediate steps to learn to produce correct final answers — what matters is the policy improvement trajectory, not the fidelity of individual traces. The quality of the pre-trained model sets the floor for effective bootstrapping, but the tolerance for noisy intermediates is built into the convergence guarantee.

Inquiring lines that read this note 291

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

Why do benchmark improvements fail to reflect actual reasoning quality? How do prompt structure and constraints affect model instruction reliability? Do corrupted reasoning traces serve as effective supervision signals? When do additional thinking tokens stop improving reasoning performance? Can prompting inject entirely new knowledge into language models? Why do reasoning models fail at systematic problem-solving and search? How do training data properties shape reasoning capability development? Does self-reflection enable models to reliably correct their errors? Do base models contain latent reasoning that training can unlock? What capability tradeoffs emerge when scaling model reasoning abilities? Why do correct reasoning traces tend to be shorter than incorrect ones? What structural advantages do diffusion language models offer over autoregressive methods? Can next-token prediction alone produce genuine language understanding? Do reasoning traces faithfully represent or merely mimic actual model reasoning? How should models express uncertainty rather than forced confident answers? What actually drives chain-of-thought reasoning improvements in language models? Why does training format shape reasoning strategy more than domain content? Do language models perform faithful symbolic reasoning independent of semantic grounding? Does recurrence enable reasoning capabilities that fixed-depth transformers cannot achieve? How does latent reasoning compare to verbalized chain-of-thought? How effectively do deterministic tools improve language model reasoning on formal tasks? How does example difficulty affect learning efficiency in language models? How can process reward models supervise complex reasoning traces? How can AI systems learn from failures without cascading errors? What factors beyond surface content determine how readers extract meaning differently? Why does supervised fine-tuning improve accuracy while degrading reasoning quality? Why does self-revision increase model confidence while degrading accuracy? How do knowledge graphs enable efficient multi-hop reasoning over alternatives? How should inference compute be adaptively allocated based on prompt difficulty? Can prompting strategies overcome LLM biases without model fine-tuning? How does memorization interact with learning and generalization? What distinguishes dynamic from static grounding in dialogue systems? How do LLMs distinguish causal reasoning from temporal and semantic associations? How does reasoning graph topology affect breakthrough insights and generalization? What memory architectures best support persistent reasoning across extended interactions? How can models identify insufficient information and respond appropriately without guessing? How should agents balance memory condensation to optimize context efficiency? What limits mechanistic interpretability's ability to characterize models? How does sequence length affect sparsity tolerance in models? Why do language models struggle with implicit discourse relations? Can AI-generated outputs constitute genuine knowledge or valid claims? Can inference-time compute substitute for scaling up model parameters? Do language models understand semantics or rely on pattern matching? Why does finetuning cause catastrophic forgetting of model capabilities? Do language models develop causal world models or rely on statistical patterns? Does tokenized intelligence retain genuine value through exchange-based systems? What dimensions of recommendation quality do standard metrics miss? How do training priors constrain what context information can override? Does fine-tuning modify underlying model capabilities or only behavioral outputs? How faithfully do LLMs reflect their actual reasoning in outputs and explanations? Why do agents confidently report success despite actually failing tasks? What are the consequences of models training on synthetic data? How do soft continuous representations explore multiple reasoning paths simultaneously? How do neural networks separate factual knowledge from reasoning abilities? What mechanisms drive sycophancy and how can we mitigate it? How can humans calibrate appropriate trust in AI systems? Does reinforcement learning teach reasoning or just when to reason? What properties determine whether reward signals teach genuine reasoning? Does alignment training create blind spots in detecting genuine safety threats?

Related concepts in this collection 8

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
22 direct connections · 134 in 2-hop network ·medium cluster Open in graph ↗
Do reasoning traces need to be semantically corr… Do reasoning traces actually cause correct answers… Does chain-of-thought reasoning reveal genuine inf… Do language models actually use their reasoning st… Can minimal reasoning chains match full explanatio… Does training on messy search processes improve re… What do models actually learn from chain-of-though… Does logical validity actually drive chain-of-thou… What three separate factors drive chain-of-thought…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

Can structural perturbations harm model accuracy more than semantic ones?Does the DeepSeek R1 single token insertion represent genuine reasoning?What computational role do intermediate tokens actually play in transformers?Can external verifiers replace reasoning trace quality in solution guarantees?
Original note title

deliberately corrupted reasoning traces perform comparably to correct traces and sometimes generalize better out of distribution