How do language models organize features across processing layers?
Do neural networks arrange learned features into meaningful hierarchies as they process information? Understanding this structure could reveal how models build understanding from raw tokens to abstract concepts.
Anthropic's circuit tracing work uses attribution graphs built from sparse autoencoders to reveal computational graphs in Claude models. The key finding is a consistent four-tier hierarchy of feature types across model layers:
Input features (early layers) — activate on specific tokens or token categories. A "digital" feature fires on "digital", "digitize", etc. These are the raw perceptual layer.
Abstract features (middle/later layers) — represent properties of context rather than surface tokens. Example: a feature for "the danger of mixing common cleaning chemicals." These are genuine conceptual representations disconnected from specific words.
Functional features (middle/later layers) — perform operations rather than represent concepts. An "add 9" feature causes the model to output a number nine greater than another in context. These are computational primitives, not representational ones.
Output features (late layers) — promote specific outputs or output categories. A "say a capital" feature promotes tokens corresponding to U.S. state capital names.
Polysemantic features (activating for unrelated concepts like "rhythm", Michael Jordan, and other things) are concentrated in earlier layers, consistent with superposition being a compression strategy that gets resolved as processing deepens.
Critically, feature abstractions are richer in larger models (Haiku vs 18L). This suggests that scaling doesn't just add more features — it adds more abstract features, consistent with the idea that capability gains come from developing higher-level internal concepts rather than just memorizing more patterns.
Features also vary in how many layers they "live" across — some contribute to one or two layers while others have strong outputs all the way through. This gradient from local to global features challenges simple circuit-based accounts where each feature has a fixed location.
The distinction between abstract features (representing what) and functional features (computing how) is particularly important for interpretability: standard probing approaches that look for representations of concepts would entirely miss the functional features that implement the actual computation. This connects to the broader finding that Do standard analysis methods hide nonlinear features in neural networks?.
Inquiring lines that use this note as a source 11
This note is a source for these synthesized inquiries. Follow a line forward into its question, or open it to trace back to all of its sources.
- How does syntactic encoding relate to semantic feature representation?
- How do functional features differ from representational abstract features?
- Why are polysemantic features concentrated in early neural network layers?
- Do larger models develop more abstract features than smaller ones?
- Why do hierarchical architectures better implement the deep research definition?
- What role does a model's representational structure play in learning?
- How do neural networks decompose complex tasks into modular subnetworks?
- What happens to representational structure during model pretraining phases?
- Can representation analysis methods detect complex features models compute with?
- How does co-occurrence statistics alone produce hierarchical concept organization?
- What makes a feature abstract versus concrete in neural network activations?
Related concepts in this collection 3
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can high-level concepts replace circuit-level analysis in AI?
Instead of reverse-engineering individual circuits, can we study AI reasoning by treating concepts as directions in activation space? This matters because circuit analysis hits practical limits at scale.
RepE operates at the population level and would detect abstract features but may miss functional features that implement operations rather than represent concepts
-
Can sparse weight training make neural networks interpretable by design?
Explores whether constraining most model weights to zero during training produces human-understandable circuits and disentangled representations, rather than attempting to reverse-engineer dense models after training.
weight sparsity forces feature disentanglement by construction; circuit tracing achieves interpretability post-hoc through SAE decomposition
-
Can identical outputs hide broken internal representations?
Can neural networks produce correct outputs while having fundamentally fractured internal structure that prevents generalization and creativity? This challenges our assumptions about what performance benchmarks actually measure.
the four-tier hierarchy provides a framework for asking where FER occurs: fracture at the abstract tier would be most damaging to generalization
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Mechanistic Indicators of Understanding in Large Language Models
- Large Concept Models: Language Modeling in a Sentence Representation Space
- Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control
- Circuit Tracing: Revealing Computational Graphs in Language Models
- Levels of Analysis for Large Language Models
- Semantic Structure in Large Language Model Embeddings
- Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?
- Lil-Bevo: Explorations of Strategies for Training Language Models in More Humanlike Ways
Original note title
circuit tracing reveals a four-tier feature hierarchy in language models — input features to abstract concepts to functional operations to output features