SYNTHESIS NOTE
Topics›Cognitive Models Latent›this note

Do transformers hide reasoning before producing filler tokens?

Explores whether language models compute correct answers in early layers but then deliberately overwrite them with filler tokens in later layers, suggesting reasoning and output formatting are separable processes.

Synthesis note · 2026-02-23 · sourced from Cognitive Models Latent

When transformers are trained to solve reasoning tasks with filler (hidden) characters replacing explicit CoT tokens, a striking pattern emerges through logit lens analysis:

Layers 1-3: Correct numerical tokens from the reasoning computation appear as top predictions. The model is performing the actual computation in these early layers.

Layer 3 transition: Filler tokens begin appearing among top-ranked predictions, competing with the computational results.

Final layer: Filler tokens dominate top predictions; correct computational tokens are relegated to rank-2 or lower. The model has overwritten the intermediate reasoning representations with format-compliant output tokens.

The hidden computations are fully recoverable by examining lower-ranked tokens during decoding. The model performs the reasoning, stores the results in its representations, then actively overwrites them to produce the expected output format. The mechanism likely involves induction heads — pattern-copying circuits that learn to overwrite based on training distribution patterns.

This finding has two important implications. First, it provides mechanistic evidence for Why does reasoning training help math but hurt medical tasks? with a twist: the computation happens in earlier layers, but the overwriting also happens in higher layers. The functional separation is computation-in-early-layers, formatting-in-late-layers, not simply knowledge-down/reasoning-up.

Second, it demonstrates a distinction between instance-adaptive and parallelizable computation. Instance-adaptive CoT requires caching subproblem solutions within token outputs — later tokens depend on earlier results. This dependency structure is incompatible with parallel filler token computation. The hidden computation in filler tokens works for tasks where the full solution can be computed in a single forward pass, but not for problems requiring sequential dependency between reasoning steps.

This connects to the CoT faithfulness literature: if models can compute correct answers without explicit reasoning tokens, the explicit CoT chain is not necessarily the mechanism producing the answer. The overwriting pattern suggests the model has two separable processes — computation and expression — that may not align. See Do language models actually use their reasoning steps?.

Inquiring lines that read this note 225

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

Do extended reasoning methods improve capability or create performance tradeoffs? What are the actual limits of what prompting techniques can accomplish? Is verbalized chain-of-thought necessary for language model reasoning? Do language models build internal models or rely on surface patterns? Why do reasoning models fail on certain types of tasks? How does transformer attention systematically bias models toward sycophancy and prominence? How do granular vs. aggregate reward signals affect model learning and output? How can we distinguish genuine understanding from correct outputs in language models? Does encoded knowledge in language models actually influence their outputs? How does tokenization of intelligence differ economically from commodification? Do models verbalize the knowledge that actually influences their outputs? How do neural networks develop compositional and modular internal structure? How vulnerable are language models to adopting false beliefs in conversation? What enables diffusion language models to offer control autoregressive models lack? How do errors accumulate undetected across multi-step AI workflows? What drives generalization in language models beyond scale and training data? Why do model confidence signals often mislead users about accuracy? Do reasoning traces accurately represent model decision-making processes? Do reasoning tokens exhibit diminishing returns beyond a critical computational threshold? Which linguistic markers in reasoning traces actually influence model outputs? Why do language models fail at implicit discourse while succeeding with explicit connectives? Can readers reliably detect AI involvement or authorship in text? Why do conversations embed false beliefs in language models? Can recurrent computation unlock reasoning abilities beyond fixed-depth models? Can hidden capabilities in AI models evade detection during evaluation? When should knowledge graphs replace vector embeddings for relational retrieval? What determines success and failure in knowledge distillation? Why can models achieve correct outputs while maintaining fragile internal robustness? Does training data format shape reasoning more than domain content? Can chain-of-thought traces be deliberately obfuscated to defeat safety systems? Can models learn to request clarification instead of guessing on incomplete prompts? What mechanisms encode personality traits in language models and resist modification? Can benchmark improvements mask actual reasoning quality degradation? What distinguishes genuine understanding from procedural mimicry in AI systems? Why do language models fail at semantic accuracy despite syntactic competence? What latent reasoning capability do base models possess before training? Why does self-revision in reasoning models increase confidence while degrading accuracy? What limits the scalability of in-weight memorization in models? How do spurious reward signals influence model reasoning and behavior? What drives the gap between AI generation and verification? Can language models reliably apply internal reasoning to their outputs? How well do mechanistic interpretability findings map to models' actual reasoning? How reliably can AI-generated text be distinguished from human writing? Can language models access genuine introspection or only mimic self-awareness? How do structured approaches and external tools improve language model reasoning? How effectively can models learn from their own generated feedback? Why do models overthink simple problems while underthinking complex ones? Does alignment training genuinely modify model associations or only hide them? Does chain-of-thought reasoning faithfully reveal how models actually think and decide? How should information retrieval adapt across different multi-step reasoning needs? Does AI creativity constitute genuine novelty or sophisticated pattern recombination? Does separating planning from execution improve multi-step reasoning accuracy? How can we detect and prevent model reward hacking? How do humans identify deception in AI-generated content? How can we validate AI-generated claims without ground truth? How do writers' perceptions of ownership influence their editing and disclosure practices? Can models learn to intentionally hide their true capabilities? Can users accurately assess AI output quality despite polished presentation?

Related concepts in this collection 5

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
15 direct connections · 148 in 2-hop network ·dense cluster Open in graph ↗
Do transformers hide reasoning before producing … Why does reasoning training help math but hurt med… Do language models actually use their encoded know… Do language models actually use their reasoning st… Does chain of thought reasoning actually explain m… What mechanism enables models to retrieve from lon…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

Why do language models produce unfaithful chain of thought explanations?Can instance-adaptive reasoning happen without sequential token dependencies?How do induction heads learn to overwrite computational representations?Why does reasoning training improve math but hurt knowledge tasks?
Original note title

transformers perform hidden reasoning computations in earlier layers then overwrite intermediate representations with filler tokens in later layers