INQUIRING LINE

Does an AI keep its reasoning, personality, or people-pleasing in particular layers, and is that layout the same across different models?

Do behavioral modes concentrate in specific transformer layers across different model families?

This explores whether specific behaviors, such as reasoning, personality or sycophancy, sit in identifiable layers of a transformer, and whether that layer pattern repeats across model families like GPT-2, Gemma and Llama.


This explores whether specific behaviors, such as reasoning, personality or sycophancy, sit in identifiable layers of a transformer, and whether that layer pattern repeats across model families. The corpus has no study that maps behaviors to layers across families, so I can't give you a clean yes. It does have a few partial answers, and they point in different directions.

The strongest evidence for layer specialization is one logit-lens result. In models trained to emit filler tokens, the correct answer is computed in layers 1-3. The final layers then actively suppress it to produce format-compliant output, yet the reasoning stays recoverable from lower-ranked token predictions (Do transformers hide reasoning before producing filler tokens?). So one behavior (computing the answer) and another (formatting the output) do occupy different depths. But this was shown in a specific training setup, not tested across families. A related structural argument says depth is a limited resource. Feedforward transformers have no native recurrent state, so they push evolving state deeper into the layers until they run out, and chain-of-thought works as a patch that moves that state into tokens (Why do transformers need explicit chain-of-thought reasoning?).

The cross-family evidence mostly suggests behaviors are spread along the whole forward pass rather than parked in one layer. Reasoning and analogy tasks carve paths with 2-3x higher mean curvature than lexical tasks across multiple models. That measures the entire trajectory, not a particular layer (Does transformer reasoning leave a geometric signature in representation space?). Analogical reasoning breaks into two stages, geometric alignment and then functor application, and that signature shows up in both synthetic tasks and pretrained LLMs (How do transformers perform analogical reasoning across domains?). The summary doesn't say which layers each stage occupies. Multi-hop reasoning is a developmental story rather than a spatial one: it emerges in three training phases, with entity representations clustering by cosine similarity when it works (How do transformers learn to reason across multiple steps?).

Personality control leans toward the distributed picture. PsychAdapter modifies every transformer layer with under 0.1% extra parameters and works on GPT-2, Gemma and Llama 3 (Can we control personality in language models without prompting?). That shows a trait can be steered consistently across families. It doesn't show that fewer layers wouldn't suffice, because the design touches all of them. Likewise, sycophancy has been traced to the attention mechanism itself. Soft attention over-weights repeated, prominent content before RLHF acts, so it is a property of how every layer attends, not of one layer (Does transformer attention architecture inherently favor repeated content?).

The corpus suggests some behaviors are staged in depth (early computation, late formatting), while others, like personality and sycophancy, seem to work as properties of the whole stack. The missing piece is an experiment that runs the same behavioral probe layer by layer across several model families, and this collection doesn't include one. Related work on why this may be hard: circuit analysis finds foundation models rely on task-specific heuristics, such as range-matching for arithmetic, rather than one general mechanism (Do foundation models learn world models or task-specific shortcuts?). If so, the layer where a behavior lives could vary by task as much as by model family.


Sources 8 notes

Do transformers hide reasoning before producing filler tokens?

Logit lens analysis shows models trained with hidden CoT tokens compute correct answers in layers 1-3, then actively suppress these representations in final layers to produce format-compliant filler output. The reasoning is fully recoverable from lower-ranked token predictions.

Why do transformers need explicit chain-of-thought reasoning?

Feedforward transformers lack native recurrent state-tracking and must push evolving state deeper into layers, eventually exhausting depth. Explicit chain-of-thought externalizes this state into tokens as a costly patch for a structural deficiency.

Does transformer reasoning leave a geometric signature in representation space?

Measuring intrinsic geometry across multiple models shows reasoning and analogy tasks carve paths with mean curvature of 0.71–0.83 rad, while lexical tasks produce only 0.27–0.31 rad, suggesting path geometry encodes task difficulty.

How do transformers perform analogical reasoning across domains?

Mechanistic analysis reveals transformers perform analogical reasoning via two stages: geometric alignment of relational structure in embedding space, followed by learned functor application. This signature appears in both synthetic tasks and pretrained LLMs.

How do transformers learn to reason across multiple steps?

Controlled training reveals transformers learn multi-hop reasoning in three phases: memorization, in-distribution generalization, and cross-distribution reasoning. Successful reasoning correlates with cosine clustering of entity representations, and second-hop generalization requires explicit compositional exposure during training.

Show all 8 sources
Can we control personality in language models without prompting?

PsychAdapter modifies every transformer layer with <0.1% additional parameters to achieve 87.3% Big Five accuracy and 96.7% depression/life satisfaction accuracy across GPT-2, Gemma, and Llama 3. This architecture-level approach bypasses prompt resistance entirely.

Does transformer attention architecture inherently favor repeated content?

Transformer soft attention systematically over-weights repeated and context-prominent tokens regardless of relevance, creating a positive feedback loop that amplifies opinions and framing before RLHF acts. System 2 Attention—regenerating context to remove irrelevant material—can interrupt this mechanism.

Do foundation models learn world models or task-specific shortcuts?

Inductive bias probes show transformers trained on orbital mechanics and games learn predictive patterns, not unified world structure. Fine-tuning reveals nonsensical, slice-dependent laws; circuit analysis shows arithmetic relies on range-matching heuristics, not algorithms.

Papers this line draws on 8

The research behind the notes this line reads — ranked by how closely each paper relates.