SYNTHESIS NOTE
Topics›Cognitive Models Latent›this note

Do transformers hide reasoning before producing filler tokens?

Explores whether language models compute correct answers in early layers but then deliberately overwrite them with filler tokens in later layers, suggesting reasoning and output formatting are separable processes.

Synthesis note · 2026-02-23 · sourced from Cognitive Models Latent

When transformers are trained to solve reasoning tasks with filler (hidden) characters replacing explicit CoT tokens, a striking pattern emerges through logit lens analysis:

Layers 1-3: Correct numerical tokens from the reasoning computation appear as top predictions. The model is performing the actual computation in these early layers.

Layer 3 transition: Filler tokens begin appearing among top-ranked predictions, competing with the computational results.

Final layer: Filler tokens dominate top predictions; correct computational tokens are relegated to rank-2 or lower. The model has overwritten the intermediate reasoning representations with format-compliant output tokens.

The hidden computations are fully recoverable by examining lower-ranked tokens during decoding. The model performs the reasoning, stores the results in its representations, then actively overwrites them to produce the expected output format. The mechanism likely involves induction heads — pattern-copying circuits that learn to overwrite based on training distribution patterns.

This finding has two important implications. First, it provides mechanistic evidence for Why does reasoning training help math but hurt medical tasks? with a twist: the computation happens in earlier layers, but the overwriting also happens in higher layers. The functional separation is computation-in-early-layers, formatting-in-late-layers, not simply knowledge-down/reasoning-up.

Second, it demonstrates a distinction between instance-adaptive and parallelizable computation. Instance-adaptive CoT requires caching subproblem solutions within token outputs — later tokens depend on earlier results. This dependency structure is incompatible with parallel filler token computation. The hidden computation in filler tokens works for tasks where the full solution can be computed in a single forward pass, but not for problems requiring sequential dependency between reasoning steps.

This connects to the CoT faithfulness literature: if models can compute correct answers without explicit reasoning tokens, the explicit CoT chain is not necessarily the mechanism producing the answer. The overwriting pattern suggests the model has two separable processes — computation and expression — that may not align. See Do language models actually use their reasoning steps?.

Inquiring lines that read this note 228

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

Does chain-of-thought reasoning faithfully reflect actual model reasoning processes? Does prompting inject new knowledge into models or activate existing knowledge? How do transformer circuits and attention mechanisms enable or limit reasoning? Do language models learn underlying linguistic structure or merely surface patterns? What determines whether visible reasoning traces improve task performance? How do different post-training structures affect learning outcomes? What can prompting techniques actually accomplish in language models? How does tokenizing intelligence reshape labor markets and institutional outcomes? How do mechanistic features and representations drive model behavior? Why do language models systematically fail at causal reasoning? What reasoning limitations and biases do language models systematically exhibit? When do additional thinking tokens stop improving reasoning performance? How do diffusion language models compare to autoregressive architectures? Do language models truly understand their encoded knowledge or merely retrieve patterns? Does polished AI output mislead users about actual accuracy and expertise? Do reasoning traces accurately represent what models actually compute? What makes reasoning models struggle with ill-posed and unfamiliar problems? Why do users believe false claims despite fact-checking efforts? What causes foundation models to develop task heuristics instead of understanding? Can prompts override training-embedded biases in language models? How do pretraining data composition and task diversity shape reasoning? How can parameter sparsity improve interpretability while maintaining model capability? Does training data format shape reasoning more than domain content? How can we distinguish genuine model self-knowledge from behavioral performance? How much do stated privacy concerns predict actual privacy-protective behavior? Does reasoning architecture matter more than model scale? How do prompt design choices and framing measurably influence model outputs? Does reasoning fine-tuning trade off other critical model abilities? How does verifier integration affect the generation-verification gap? How does AI writing assistance affect reader perception of writer authenticity? Should reward models emphasize process understanding or only final outcomes? What explains the gap between LLM semantic understanding and production? How do dialogue structure and state management affect multi-turn reasoning quality? Why does self-revision increase model confidence despite degraded accuracy? What drives sycophancy in language models and can it be fixed? Is model misalignment inherent to base models or induced by alignment procedures? What actually drives chain-of-thought performance gains in reasoning tasks? What fundamental limitations prevent embeddings from solving task-relevant retrieval? Does generative AI replace coordination with individual productivity? Can we reliably detect when models engage in reward hacking? How does transparency of agent reasoning affect effective oversight? Can we reliably distinguish AI-generated from human-written text? Can models strategically underperform to evade capability evaluation? Do AI explanations represent actual model reasoning or constructed post-hoc narratives? Does AI-assisted coding actually improve developer productivity and code quality?

Related concepts in this collection 5

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
15 direct connections · 148 in 2-hop network ·dense cluster Open in graph ↗
Do transformers hide reasoning before producing … Why does reasoning training help math but hurt med… Do language models actually use their encoded know… Do language models actually use their reasoning st… Does chain of thought reasoning actually explain m… What mechanism enables models to retrieve from lon…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

Why do language models produce unfaithful chain of thought explanations?Can instance-adaptive reasoning happen without sequential token dependencies?How do induction heads learn to overwrite computational representations?Why does reasoning training improve math but hurt knowledge tasks?
Original note title

transformers perform hidden reasoning computations in earlier layers then overwrite intermediate representations with filler tokens in later layers