Why do transformers need explicit chain-of-thought reasoning?
Explores whether chain-of-thought is a fundamental reasoning mechanism or a workaround for architectural limitations in how transformers track evolving state across computation steps.
The argument here is structural, not empirical, and it recasts a lot of the reasoning literature. State tracking — iteratively updating latent variables as an environment evolves, s_t = f(s_{t-1}, x_t) — is inherently sequential. A purely feedforward transformer cannot perform that update in place: with each new input step it must push the evolving state representation deeper into its layer stack, which renders earlier state inaccessible in shallow layers and eventually exhausts the model's finite depth. From this view, the entire apparatus of explicit chain-of-thought and latent "thinking" is not the mechanism of reasoning but a workaround — it externalizes state into the token stream because the architecture cannot hold it internally. The proposed fix is to refocus from explicit thought traces to implicit recurrent activation dynamics, with a taxonomy organized by recurrence axis (depth vs step) and the ratio of input tokens to recurrence steps.
This is the theoretical spine for the vault's recurrence cluster. How do looped language models actually improve reasoning in depth? gives the mechanistic picture of what depth-axis recurrence is doing; Can tiny recursive networks outperform massive language models? is the existence proof that recursion on latent state beats scale on exactly the state-heavy tasks this paper predicts will exhaust feedforward depth. Can looped transformers generalize to unseen knowledge combinations? supplies the "cannot" — a capability gap closed only by recurrence.
The counterargument the paper must answer is Can state-space models match transformers at copying and retrieval?: recurrent fixed-size state has its own provable ceiling on copying and retrieval. So the honest synthesis is not "recurrence beats attention" but a division of labor — attention's expanding context is right for retrieval, recurrence is right for state tracking, and conflating the two is what makes both CoT externalization and pure SSMs disappoint. The provocative line for writing: the field has been paying a token tax to simulate a state-update operation the hardware should perform natively.
Inquiring lines that use this note as a source 7
This note is a source for these synthesized inquiries. Follow a line forward into its question, or open it to trace back to all of its sources.
- Can recurrent transformers track state more efficiently than feedforward models?
- How do transformers compare to state-space models on copying and retrieval?
- Do looped transformers naturally converge to fixed points during inference?
- How do parallel loops with position offsets differ from sequential loop architectures?
- Why does explicit chain-of-thought work as a workaround for feedforward transformers?
- How much of chain-of-thought reasoning is actually redundant?
- Which problems cannot be solved by parallel architectures and require serial depth?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
How do looped language models actually improve reasoning in depth?
Mechanistic analysis investigates whether looping transformer layers creates genuinely new computation or reuses existing inferential stages. Understanding this distinction clarifies why recurrent depth can match standard scaling.
grounds (mechanism of depth-axis recurrence the taxonomy categorizes)
-
Can tiny recursive networks outperform massive language models?
Can a small network that recursively refines its reasoning on a latent state match or beat billion-parameter LLMs on hard reasoning puzzles? This challenges assumptions about scale and hierarchy in AI reasoning.
exemplifies (recursion on latent state beats scale on state-heavy tasks)
-
Can looped transformers generalize to unseen knowledge combinations?
Do transformers that reuse layers across iterations succeed where standard transformers fail at composing facts in novel ways? This matters because systematic generalization is a hallmark of human reasoning.
exemplifies (capability gap closed only by recurrence)
-
Can state-space models match transformers at copying and retrieval?
Explores whether the efficiency gains of state-space models come at a fundamental cost in their ability to copy strings and retrieve exact information from context, compared to transformers.
contradicts (recurrent fixed-size state has its own provable ceiling — forces a division-of-labor reading)
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- The Topological Trouble With Transformers
- A Mechanistic Analysis of Looped Reasoning Language Models
- Implicit Chain of Thought Reasoning via Knowledge Distillation
- Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety
- Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization
- How do Transformers Learn Implicit Reasoning?
- Compositional Reasoning with Transformers, RNNs, and Chain of Thought
- Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens
Original note title
explicit chain-of-thought is an inefficient workaround for a topological deficiency — feedforward transformers cannot track evolving state without recurrence