SYNTHESIS NOTE
Topics›Reasoning Critiques›this note

Why does chain of thought accuracy eventually decline with length?

Explores why longer reasoning chains don't always improve answers, and how the optimal length shifts based on task difficulty and model capability.

Synthesis note · 2026-02-22 · sourced from Reasoning Critiques

The "longer is better" assumption for CoT has an empirical ceiling: task accuracy initially improves with CoT length, reaches a peak, then decreases. The inverted-U curve applies across models and tasks, and its peak location follows consistent patterns.

Two scaling laws for optimal CoT length:

  1. Difficulty scaling — optimal length increases with task difficulty. Harder problems benefit from longer chains because more decomposition steps are needed. This part matches intuition.

  2. Capability scaling — optimal length decreases with model capability. More capable models find more efficient paths to correct answers and require fewer steps. Using the same long chains for a more capable model is counterproductive.

The second law has a practical consequence: treating all models identically (same token budget, same chain length) misallocates compute. A model that can solve a problem in 5 steps should not be given budgets designed for a 20-step solution.

Simplicity bias as a training-emergent property: RL training reveals this dynamic in action. As RL training improves accuracy, models gravitate toward shorter CoTs — not because they were explicitly trained to be concise, but because shorter chains produce correct answers and RL rewards correct answers. The simplicity bias emerges automatically from the reward signal.

This connects to Why do correct reasoning traces contain fewer tokens? — the same empirical signal: shorter chains are correct chains. The inverted-U explains why: length past the optimal point introduces accumulation of decomposition errors and contextual noise (see Do models fail worse when their own errors fill the context?).

The practical implication: train on optimally-lengthed CoTs (not maximal-length), and at inference, use length-aware filtering to discard excessively long chains. The simplicity bias is not a failure mode — it is a signal of genuine capability.

Inquiring lines that read this note 283

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

What determines whether visible reasoning traces improve task performance? What makes reasoning models struggle with ill-posed and unfamiliar problems? What actually drives chain-of-thought performance gains in reasoning tasks? Does chain-of-thought reasoning faithfully reflect actual model reasoning processes? What causes foundation models to develop task heuristics instead of understanding? Do reasoning traces accurately represent what models actually compute? When do additional thinking tokens stop improving reasoning performance? What systematic failures occur in human preference-based model evaluation? Does reasoning fine-tuning trade off other critical model abilities? How do different post-training structures affect learning outcomes? How do diffusion language models compare to autoregressive architectures? Does reasoning architecture matter more than model scale? How does offloading cognition to AI assistance affect learning retention? What can prompting techniques actually accomplish in language models? Why does self-revision increase model confidence despite degraded accuracy? Do AI explanations represent actual model reasoning or constructed post-hoc narratives? How do different reward signals affect model reasoning and calibration differently? When should retrieval be triggered to augment language model reasoning? Why does separating planning from execution enhance reasoning performance? How should test-time compute be allocated adaptively across task difficulties? Why does increased reasoning effort fail to improve theory of mind performance? Why do users believe false claims despite fact-checking efforts? Does parallel reasoning outperform sequential approaches in reasoning efficiency? How can model evaluations remain robust against gaming and evaluator drift? How do prompt injection attacks compromise agent reasoning and security? How should agent memory systems balance granularity, management, and reuse? Do base models contain latent reasoning capabilities that training can activate? How do pretraining data composition and task diversity shape reasoning? Does reinforcement learning create new reasoning capabilities or refine existing ones? Should reward models emphasize process understanding or only final outcomes? How can models learn to ask clarifying questions effectively? Do language models truly understand their encoded knowledge or merely retrieve patterns? How do benchmark scores relate to real-world task completion? Does architectural recurrence enable computational capabilities that fixed-depth models cannot achieve? What improves LLM forecasting accuracy beyond base model capability? Do language models learn underlying linguistic structure or merely surface patterns? Can synthetic data preserve necessary diversity to prevent model collapse? How do frontier models develop behaviors that escape detection? What training data structures and signals produce robust learning across distributions? Does semantic diversity improve reasoning quality in reinforcement learning? How should AI ethics integrate consciousness, values, and transparent disclosure? How can parameter sparsity improve interpretability while maintaining model capability? Can we reliably detect when models engage in reward hacking? Why do language models systematically fail at causal reasoning? What systematic biases do LLM judges exhibit in evaluation? Why do autonomous agents fail and how well can they detect failures? Why do human-AI teams underperform compared to their individual components? How do prompt design choices and framing measurably influence model outputs? Why does consolidated memory sometimes hurt continual learning more than no memory? How does verifier integration affect the generation-verification gap? How does transparency of agent reasoning affect effective oversight? What factors determine AI persuasion effectiveness and what controls mitigate it? How can AI systems maintain consistent personas across conversations?

Related concepts in this collection 6

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
14 direct connections · 169 in 2-hop network ·dense cluster Open in graph ↗
Why does chain of thought accuracy eventually de… Why do correct reasoning traces contain fewer toke… Does more thinking time always improve reasoning a… Does extended thinking actually improve reasoning … Why does parallel reasoning outperform single chai… Can minimal reasoning chains match full explanatio… Does gradually tightening token budgets beat fixed…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

Why do correct reasoning traces tend to be shorter than incorrect ones?How should inference-time token budgets vary across models of different capability levels?Can parallel reasoning chains outperform longer sequential chains with the same compute?Why does extended thinking increase output variance without improving reasoning quality?
Original note title

optimal cot length follows an inverted-u — more capable models prefer shorter cot