What makes a feature abstract versus concrete in neural network activations?
This explores what actually distinguishes an 'abstract' feature from a 'concrete' one inside a network's activations — and the corpus suggests the dividing line is less about the feature itself than about how deep into the network's processing it sits and how the model got there.
This explores what makes a feature count as abstract rather than concrete inside neural activations. The cleanest answer in the corpus comes from circuit tracing in Claude-family models, which finds features arranged in four tiers: raw token-level inputs at the bottom, then abstract concepts, then functional operations, then outputs How do language models organize features across processing layers?. On this view, 'concrete' and 'abstract' aren't two kinds of thing — they're positions on a processing gradient. A concrete feature stays close to the surface form of the input (this token, this string); an abstract feature has been lifted away from any particular surface realization into a concept the model reuses across many inputs. Tellingly, larger models grow richer abstract tiers, which is read as evidence that scale buys higher-level conceptual reasoning rather than just more memorized patterns.
Why would abstraction emerge from depth at all? One answer is mathematical: predicting your own latent representations recovers compositional, hierarchical structure exponentially faster than predicting raw tokens, because features at the same level of abstraction are far more correlated with each other than the noisy tokens beneath them Why is predicting latents more sample-efficient than tokens?. Abstraction, in other words, is where the learnable signal lives. The same theme shows up in how networks naturally carve compositional tasks into isolated modular subnetworks, each implementing a reusable subroutine — abstraction as the reuse of a learned piece across novel combinations Do neural networks naturally learn modular compositional structure?, an ability that strengthens with scale and broad training coverage Can neural networks learn compositional skills without symbolic mechanisms?.
But here's the part you might not expect: a feature being abstract is not the same as it being well-organized, and our usual tools quietly confuse the two. Standard analysis methods — PCA, linear regression, RSA — systematically over-report simple linear features and under-report the nonlinear ones, to the point where a network can compute perfectly with activation structure that looks like noise to every interpretability tool we have Do standard analysis methods hide nonlinear features in neural networks?. So 'concrete vs abstract' as humans label it may partly be an artifact of what our instruments can see, not a property the network respects. The 'abstract' features we celebrate finding may be the ones that happened to land in a linearly decodable form.
That caveat sharpens when you look at fractured representations. Two networks can produce identical outputs yet have radically different internal organization — one clean and reusable, the other entangled and brittle — and weight-perturbation reveals the difference where accuracy never would Can identical outputs hide broken internal representations?. A feature can be linearly decodable (looks abstract, looks clean) while the underlying organization is broken, leaving the model fragile to distribution shift Can models be smart without organized internal structure?. Real abstraction — the kind that transfers and recombines — is exactly what the binding problem says distributed networks struggle to maintain: segregating entities, keeping them representationally separate, and reusing them in new combinations Why do neural networks fail at compositional generalization?.
The thing you didn't know you wanted to know: even how dense or sparse a feature's activation is turns out to be learned, not fixed. Networks fire densely for familiar training data and fall back to sparse representations for unfamiliar inputs Is representational sparsity learned or intrinsic to neural networks?. So the abstract/concrete character of a feature isn't a stable property stamped into the architecture — it's a moving target shaped by what the model has seen, where in the stack you look, and whether your measuring tool can perceive nonlinear structure at all.
Sources 9 notes
Circuit tracing in Claude models reveals features progress from token-level inputs to abstract concepts to functional operations to outputs. Larger models develop richer abstract features, suggesting scaling enables higher-level conceptual reasoning rather than pattern memorization.
A formal sample-complexity analysis proves latent-level self-supervision (data2vec/JEPA style) recovers compositional structure with samples constant in hierarchy depth, while token-level learning requires exponential samples—because same-level latents are far more correlated than raw tokens.
Pruning experiments reveal that neural networks implement compositional subroutines in isolated subnetworks, with ablations affecting only their corresponding function. Pretraining substantially increases the consistency and reliability of this modular structure across architectures and domains.
Standard MLPs achieve compositional generalization through data and model scaling alone, without architectural modifications, provided the training distribution sufficiently covers combinations of task modules. Linear decodability of constituents from hidden activations reliably predicts success.
PCA, linear regression, and RSA over-represent simple linear features while under-representing equally important nonlinear features. Homomorphic encryption demonstrates that networks can compute perfectly well with no interpretable activation structure, proving representation patterns and computation can be entirely decoupled.
Networks trained with SGD reproduce outputs perfectly while having radically different internal structure than evolved networks, with weight perturbations revealing fractured, entangled representations that prevent transfer to novel contexts or creative recombination.
Models trained with SGD can contain all the linearly decodable features needed for a task while maintaining fundamentally broken internal organization. This makes them vulnerable to perturbation and distribution shift invisible to standard evaluation metrics.
Greff et al. argue that neural networks cannot dynamically bind distributed information into compositional structures due to three failures: segregating entities from inputs, maintaining representational separation, and reusing learned structure in novel combinations. Scaling can partially overcome this by enabling compositional representations to emerge.
During pretraining, neural networks develop dense activations for familiar training data and default to sparse representations for unfamiliar inputs. This trend emerges without task-specific fine-tuning and reflects how models consolidate knowledge through exposure.