SYNTHESIS NOTE
Topics›Cognitive Models Latent›this note

Do transformers hide reasoning before producing filler tokens?

Explores whether language models compute correct answers in early layers but then deliberately overwrite them with filler tokens in later layers, suggesting reasoning and output formatting are separable processes.

Synthesis note · 2026-02-23 · sourced from Cognitive Models Latent

When transformers are trained to solve reasoning tasks with filler (hidden) characters replacing explicit CoT tokens, a striking pattern emerges through logit lens analysis:

Layers 1-3: Correct numerical tokens from the reasoning computation appear as top predictions. The model is performing the actual computation in these early layers.

Layer 3 transition: Filler tokens begin appearing among top-ranked predictions, competing with the computational results.

Final layer: Filler tokens dominate top predictions; correct computational tokens are relegated to rank-2 or lower. The model has overwritten the intermediate reasoning representations with format-compliant output tokens.

The hidden computations are fully recoverable by examining lower-ranked tokens during decoding. The model performs the reasoning, stores the results in its representations, then actively overwrites them to produce the expected output format. The mechanism likely involves induction heads — pattern-copying circuits that learn to overwrite based on training distribution patterns.

This finding has two important implications. First, it provides mechanistic evidence for Why does reasoning training help math but hurt medical tasks? with a twist: the computation happens in earlier layers, but the overwriting also happens in higher layers. The functional separation is computation-in-early-layers, formatting-in-late-layers, not simply knowledge-down/reasoning-up.

Second, it demonstrates a distinction between instance-adaptive and parallelizable computation. Instance-adaptive CoT requires caching subproblem solutions within token outputs — later tokens depend on earlier results. This dependency structure is incompatible with parallel filler token computation. The hidden computation in filler tokens works for tasks where the full solution can be computed in a single forward pass, but not for problems requiring sequential dependency between reasoning steps.

This connects to the CoT faithfulness literature: if models can compute correct answers without explicit reasoning tokens, the explicit CoT chain is not necessarily the mechanism producing the answer. The overwriting pattern suggests the model has two separable processes — computation and expression — that may not align. See Do language models actually use their reasoning steps?.

Inquiring lines that read this note 198

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

What capability tradeoffs emerge when scaling model reasoning abilities? Can prompting inject entirely new knowledge into language models? Does recurrence enable reasoning capabilities that fixed-depth transformers cannot achieve? Which computational strategies best support reasoning in language models? Why do reasoning models fail at systematic problem-solving and search? How do transformer attention mechanisms implement memory and algorithmic functions? Can next-token prediction alone produce genuine language understanding? How does latent reasoning compare to verbalized chain-of-thought? How should models express uncertainty rather than forced confident answers? How do neural networks separate factual knowledge from reasoning abilities? How do soft continuous representations explore multiple reasoning paths simultaneously? Does tokenized intelligence retain genuine value through exchange-based systems? How can models identify insufficient information and respond appropriately without guessing? What limits mechanistic interpretability's ability to characterize models? Do language models learn genuine linguistic structure or just surface patterns? Why do language models reinforce false assumptions instead of correcting them? What structural advantages do diffusion language models offer over autoregressive methods? Do language models perform faithful symbolic reasoning independent of semantic grounding? Why do benchmark improvements fail to reflect actual reasoning quality? Does AI fluency substitute for verifiable accuracy in human judgment? Why do correct reasoning traces tend to be shorter than incorrect ones? How do training priors constrain what context information can override? When do additional thinking tokens stop improving reasoning performance? Do reasoning traces faithfully represent or merely mimic actual model reasoning? How can emotions function as reliable information in reasoning and cognitive systems? Do language model representations contain causally steerable task-specific features? What mechanisms enable AI systems to generate and spread false beliefs? How do training data properties shape reasoning capability development? How do adversarial and manipulative prompts attack reasoning models? How can identical external performance mask different internal representations? Why does training format shape reasoning strategy more than domain content? Is model self-awareness based on genuine introspection or pattern matching? Why do language models struggle with implicit discourse relations? Do corrupted reasoning traces serve as effective supervision signals? Can inference-time compute substitute for scaling up model parameters? What prevents language models from reliably adopting diverse personas? Why does supervised fine-tuning improve accuracy while degrading reasoning quality? How do prompt structure and constraints affect model instruction reliability? How do professional roles and expertise transform with AI-generated content? Why can't humans reliably detect AI-generated text despite measurable linguistic signatures? What properties determine whether reward signals teach genuine reasoning? How faithfully do LLMs reflect their actual reasoning in outputs and explanations? Why does verification consistently lag behind AI generation? What role does compression play in language model capability and generalization? How should conversational agents balance goal-driven initiative with user control? Why does self-revision increase model confidence while degrading accuracy? What are the consequences of models training on synthetic data? What mechanisms drive sycophancy and how can we mitigate it? What actually drives chain-of-thought reasoning improvements in language models? What structural biases does transformer attention create in language model outputs? How can AI systems learn from failures without cascading errors? Why do semantic similarity and task relevance diverge in vector embeddings? Can model confidence signals reliably improve reasoning quality and calibration? Do language models develop causal world models or rely on statistical patterns? How effectively do deterministic tools improve language model reasoning on formal tasks?

Related concepts in this collection 5

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
15 direct connections · 148 in 2-hop network ·dense cluster Open in graph ↗
Do transformers hide reasoning before producing … Why does reasoning training help math but hurt med… Do language models actually use their encoded know… Do language models actually use their reasoning st… Does chain of thought reasoning actually explain m… What mechanism enables models to retrieve from lon…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

Why do language models produce unfaithful chain of thought explanations?Can instance-adaptive reasoning happen without sequential token dependencies?How do induction heads learn to overwrite computational representations?Why does reasoning training improve math but hurt knowledge tasks?
Original note title

transformers perform hidden reasoning computations in earlier layers then overwrite intermediate representations with filler tokens in later layers