What role does a model's representational structure play in learning?
This explores how the internal shape of what a model represents — how features are organized, how dense or sparse activations are, how close it stays to its starting point — shapes whether and how it learns, rather than treating learning as just adjusting outputs.
This explores how the internal shape of what a model represents — how its features are organized, how dense or sparse its activations are, how its geometry encodes structure — shapes whether and how it learns. The corpus makes a striking case: representation isn't a byproduct of learning, it's often the thing doing the learning. Two models can produce identical outputs through completely different internal machinery, and the differences matter — improvements in one dimension like accuracy reliably degrade others like faithfulness or calibration What actually happens inside a language model?. So "learning" can't be read off behavior alone; you have to look at the structure underneath.
That structure turns out to be surprisingly organized and self-arranging. Circuit tracing shows features sorting themselves into a four-tier hierarchy — from raw tokens up through abstract concepts to functional operations — with bigger models growing richer abstract layers rather than just memorizing more patterns How do language models organize features across processing layers?. Networks spontaneously decompose compositional tasks into isolated, modular subnetworks you can ablate one at a time, and pretraining makes that modularity more reliable Do neural networks naturally learn modular compositional structure?. They even encode syntax in a structured geometry — using both distance and angle, like polar coordinates — that looks compatible with symbolic relations How do language models encode syntactic relations geometrically?. The representational scaffolding is doing real conceptual work.
The most counterintuitive thread is how density and sparsity behave. Models develop dense activations for material they know well and default to sparse ones for unfamiliar inputs — and this is learned through training-data familiarity, not baked into the architecture Is representational sparsity learned or intrinsic to neural networks?. That same sparsity isn't a malfunction: when a task drifts out of distribution, hidden states sparsify in a localized, systematic way that acts as a selective filter stabilizing performance Do language models sparsify their activations under difficult tasks?. The model is reshaping its own representation on the fly to cope with difficulty. Relatedly, models build internal mechanisms for tracking what they actually know — entity-recognition features that causally steer whether they answer or refuse Do models know what they don't know?.
What you learn also depends on what kind of representation the knowledge lives in. Reasoning generalizes because it draws on broad, transferable procedural knowledge spread across many documents, while factual recall stays brittle, tied to narrow memorized sources Does procedural knowledge drive reasoning more than factual retrieval?. And these capability types stack rather than replace each other — mechanistic work finds conceptual, world-state, and principled understanding coexisting as a patchwork, with higher-tier circuits layered over lower-tier heuristics that never fully go away Do language models understand in fundamentally different ways?.
Finally, representational structure governs whether a model can keep learning at all. Staying close to the base model — low KL drift — preserves plasticity, letting a model absorb new tasks where parameter-only methods stall and lose the ability to adapt when domains shift Does staying close to the base model preserve learning ability?. Even learning without weight updates depends on structure: in-context learning of sequential decisions needs whole trajectories from the same environment, not isolated examples, because the model leans on the structural pattern of the sequence to generalize Why do trajectories matter more than individual examples for in-context learning?. The thread running through all of this: a model learns through the shape of what it represents, and that shape is something it actively builds, reorganizes, and protects.
Sources 11 notes
Research shows that LLMs can achieve the same output through different internal mechanisms, and improvements in one dimension like accuracy reliably degrade others like faithfulness and calibration. Internal structure matters even when behavior appears identical.
Circuit tracing in Claude models reveals features progress from token-level inputs to abstract concepts to functional operations to outputs. Larger models develop richer abstract features, suggesting scaling enables higher-level conceptual reasoning rather than pattern memorization.
Pruning experiments reveal that neural networks implement compositional subroutines in isolated subnetworks, with ablations affecting only their corresponding function. Pretraining substantially increases the consistency and reliability of this modular structure across architectures and domains.
The Polar Probe shows LLMs represent syntactic type and direction through both distance and angular position between embeddings, nearly doubling accuracy over distance-only methods. This demonstrates neural networks spontaneously learn structured, symbolic-compatible geometry.
During pretraining, neural networks develop dense activations for familiar training data and default to sparse representations for unfamiliar inputs. This trend emerges without task-specific fine-tuning and reflects how models consolidate knowledge through exposure.
As task difficulty increases, LLM hidden states become substantially sparser in a localized, systematic way that correlates with task unfamiliarity and reasoning load. This sparsification acts as a selective filter stabilizing performance under OOD shift rather than a failure mode.
Sparse autoencoders revealed that language models develop causal mechanisms for detecting whether they know facts about entities. These mechanisms actively steer both hallucination and refusal behavior, and persist from base models into finetuned chat versions.
Analysis of 5 million pretraining documents shows reasoning relies on broad, transferable procedural knowledge from diverse sources, unlike factual recall which depends on narrow, document-specific memorization of target facts.
Mechanistic interpretability reveals conceptual understanding (features as directions), state-of-world understanding (factual connections), and principled understanding (compact circuits). Crucially, higher tiers coexist with lower-tier heuristics rather than replacing them, creating a patchwork of capabilities.
FST-trained models stay up to 70% closer to their base distribution than parameter-only RL, and this reduced drift preserves the model's ability to learn subsequent tasks effectively. Parameter-only approaches stall when task domains change, while low KL drift enables sustained adaptation.
In-context learning for sequential decision-making requires full or partial trajectories from the same environment level, not just isolated examples. This structural property—trajectory burstiness—allows models to generalize across vastly different tasks without weight updates.