SYNTHESIS NOTE
Topics›Reasoning Critiques›this note

Does chain-of-thought reasoning actually generalize beyond training data?

Explores whether CoT's strong performance on benchmarks reflects genuine reasoning ability or merely reflects learned patterns tied to specific distributions. Tests how CoT behaves when tasks, formats, or reasoning length shift away from training data.

Synthesis note · 2026-02-22 · sourced from Reasoning Critiques
How should we spend compute at inference time? What kind of thing is an LLM really?

Chain-of-Thought prompting performs well on in-distribution problems and fails predictably as distributional discrepancy increases. This is not a bug — it is the fundamental nature of what CoT is.

DataAlchemy experiments train LLMs from scratch in controlled environments and probe them under three distributional shift dimensions:

  1. Task distribution shift — novel tasks with unique elements or underlying logical structure not seen during training
  2. Length distribution shift — reasoning chains substantially longer or shorter than training data length range
  3. Format distribution shift — prompt formulation variations (even minor syntactic changes) that fall outside training distribution

In all three dimensions, the pattern is the same: CoT works within distribution, fails outside it. Under moderate shifts, models generate fluent yet logically inconsistent reasoning — the form holds, the logic breaks. This is the "mirage" phenomenon: outputs look like reasoning while producing wrong conclusions.

The interpretive frame: CoT reflects a structured inductive bias learned from training data, not a generalizable reasoning capability. When a test query is within this inductive bias, CoT activates the appropriate reasoning schema and produces good outputs. When the query falls outside it, the schema mismatch produces confident-sounding nonsense.

The practical implication for CoT as a plug-and-play solution: it is not. Performance on CoT benchmarks measures in-distribution capability. Extrapolating to novel tasks, unusual prompt formulations, or unusually long/short reasoning chains is unjustified. The benchmark scores do not predict performance under distribution shift.

This provides the empirical grounding for Does chain-of-thought reasoning reveal genuine inference or pattern matching? — the mirage emerges from imitation under distribution shift: the model continues imitating the form of reasoning while having no schema to produce valid content.

Inquiring lines that read this note 249

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

How faithfully do LLMs reflect their actual reasoning in outputs and explanations? How can LLM recommenders match or exceed collaborative filtering performance? Why do benchmark improvements fail to reflect actual reasoning quality? How effectively do deterministic tools improve language model reasoning on formal tasks? Do reasoning traces faithfully represent or merely mimic actual model reasoning? How does latent reasoning compare to verbalized chain-of-thought? What actually drives chain-of-thought reasoning improvements in language models? Does recurrence enable reasoning capabilities that fixed-depth transformers cannot achieve? When do additional thinking tokens stop improving reasoning performance? Why do reasoning models fail at systematic problem-solving and search? How does reasoning graph topology affect breakthrough insights and generalization? How should iterative research systems allocate reasoning per search step? What structural advantages do diffusion language models offer over autoregressive methods? Do base models contain latent reasoning that training can unlock? How do neural networks separate factual knowledge from reasoning abilities? What determines success in training models on multiple tasks? What limits mechanistic interpretability's ability to characterize models? Can inference-time compute substitute for scaling up model parameters? How do training data properties shape reasoning capability development? Why do correct reasoning traces tend to be shorter than incorrect ones? Why does training format shape reasoning strategy more than domain content? Is model self-awareness based on genuine introspection or pattern matching? What factors beyond surface content determine how readers extract meaning differently? Can single-axis benchmarks accurately predict agent deployment success? Why does supervised fine-tuning improve accuracy while degrading reasoning quality? How does example difficulty affect learning efficiency in language models? Does model scaling alone produce compositional generalization without symbolic mechanisms? How should inference compute be adaptively allocated based on prompt difficulty? Can prompting inject entirely new knowledge into language models? How do knowledge injection methods compare across cost and effectiveness? How do knowledge graphs enable efficient multi-hop reasoning over alternatives? How do LLMs distinguish causal reasoning from temporal and semantic associations? Which computational strategies best support reasoning in language models? Does parallel reasoning outperform sequential thinking under fixed compute budgets? What makes weaker teacher models effective for stronger student training? What capability tradeoffs emerge when scaling model reasoning abilities? Do language models develop causal world models or rely on statistical patterns? How can AI systems learn from failures without cascading errors? How does policy entropy collapse constrain reasoning-focused reinforcement learning? Do corrupted reasoning traces serve as effective supervision signals? How do transformer attention mechanisms implement memory and algorithmic functions? How can process reward models supervise complex reasoning traces? Does reinforcement learning teach reasoning or just when to reason? How does reasoning effort affect AI theory of mind performance? Do language models perform faithful symbolic reasoning independent of semantic grounding? Can debate mechanisms prevent silent agreement on wrong answers in multi-agent reasoning? Do language models learn genuine linguistic structure or just surface patterns? Can graph structure and relationships fundamentally improve recommendation systems? How can AI agents autonomously learn and transfer skills across tasks? Why do reward structures fail to shape long-term agent learning? Why do continual learning scenarios trigger catastrophic forgetting and interference? Does fine-tuning modify underlying model capabilities or only behavioral outputs? Can self-supervised signals enable process supervision without human annotation? Can next-token prediction alone produce genuine language understanding? Does domain specialization cause models to lose capabilities elsewhere? How should personalization be implemented to improve AI assistant effectiveness? Does alignment training create blind spots in detecting genuine safety threats?

Related concepts in this collection 4

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
15 direct connections · 130 in 2-hop network ·medium cluster Open in graph ↗
Does chain-of-thought reasoning actually general… Does chain-of-thought reasoning reveal genuine inf… Do language models actually use their reasoning st… Can models pass tests while missing the actual gra… Does training data format shape reasoning strategy…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

Why do language models imitate reasoning form without abstract inference capability?How much does prompt format shape what reasoning strategy a model uses?Can chain-of-thought explanations be both sufficient and necessary for model decisions?Do language models learn surface patterns that appear generalizable but actually fail under shift?
Original note title

cot reasoning is distribution-bounded — effectiveness degrades predictably with distributional discrepancy