SYNTHESIS NOTE
Do neural networks naturally learn modular compositional structure?
Explores whether neural networks decompose compositional tasks into distinct subroutines without explicit symbolic design. This challenges the longstanding view that neural networks are fundamentally non-compositional.
Structural compositionality is the extent to which neural networks break down compositional tasks into subroutines and implement them in modular subnetworks. The alternative: matching inputs to learned templates without task decomposition.
The evidence supports compositionality. Using model pruning to isolate subnetworks:
- Subnetworks that implement one subroutine can be identified
- Ablating a subnetwork harms its corresponding subroutine while leaving others largely intact
- This holds across multiple architectures (CNNs, transformers), tasks (vision, language), and scales
The pretraining effect: models initialized with pretrained weights more reliably produce modular subnetworks than randomly initialized models. Self-supervised pretraining appears to create internal structure that is more amenable to compositional decomposition. This suggests that the representations learned during pretraining have a modular quality that fine-tuning can exploit.
This provides empirical support against the longstanding objection that neural networks are fundamentally non-compositional. The finding: "some simple pseudo-symbolic computations might be learned directly from data using standard gradient-based optimization techniques." Explicit symbolic mechanisms may be unnecessary — gradient-based optimization discovers compositional structure when the task demands it and pretraining provides a good initialization.
The result is not perfect: "most do not exhibit perfect task decomposition." Compositionality is partial and graded, not all-or-nothing. Some architecture-task combinations show stronger structural compositionality than others.
This connects to the weight-sparsity finding: Can sparse weight training make neural networks interpretable by design? shows that enforcing sparsity produces clean decomposition. The structural compositionality paper shows that decomposition also emerges naturally, albeit imperfectly, from standard training. Sparsity amplifies a tendency that already exists.
Inquiring lines that read this note 142
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How do different post-training structures affect learning outcomes? What distinguishes genuine understanding from correct linguistic performance? Can scaling alone produce compositional generalization in neural networks?- Can world models form from aggregated partial information across training distributions?
- Why do text-to-image models fail at composing multiple concepts together?
- Does scaling model size solve compositional generalization problems?
- Does compositional generalization emerge suddenly or improve smoothly with scale?
- Does scaling data automatically produce compositional reasoning or just better feature encoding?
- What makes recursive structure different from other forms of compositional generalization?
- Do substitute networks converge differently than complement networks?
- Can scaling alone create compositional generalization without explicit binding mechanisms?
- Can granular function calling tasks learn composition from graph-sampled data?
- What role does query-level exposure play in enabling compositional generalization?
- Why does scaling data and model size improve compositional generalization?
- Why do long-context language models struggle with compositional reasoning tasks?
- What architectural alternatives can capture compositional structure beyond pooled cosine?
- How does scaling and training data enable compositional behavior without symbolic mechanisms?
- Where do neural networks still fail at compositional generalization despite scaling?
- Why does compositional reasoning fail to explain cross-domain transfer?
- What limits the extrapolation of learned operations like rotation and reflection?
- Could superposed decoding algorithms maintain multi-task representation during generation?
- Does directional knowledge failure indicate shallow pattern matching over deep representation?
- Does training on granular tasks beat training on the full function calling problem?
- What does next-token prediction tell us about compositional linguistic competence?
- Do transformers learn generalizable algorithms or instance-based patterns?
- Can symbolic mechanisms improve transformer compositional abilities?
- Could graph neural networks fundamentally outperform transformers on structured reasoning?
- Why do standard transformers fail to encode recursive structure in their hidden states?
- How does layer removal affect transformers compared to ResNets?
- How do induction heads learn to overwrite computational representations?
- How do attention patterns and circuits function as algorithmic representations?
- What makes modernized N-gram embeddings composable with transformer architectures?
- Can spline-based activations replace MLPs in transformer architectures?
- Can transformers abstract relational structure without explicit symbolic machinery?
- Can neural networks represent symbolic structures without explicit mechanisms?
- Why do human-designed neural architectures eventually get replaced by learned ones?
- Could probing methods miss computationally important features in neural networks?
- Why are polysemantic features concentrated in early neural network layers?
- Can neural networks implement genuine algorithms or only statistical pattern matching?
- Can fractured representations explain why models fail at systematic generalization?
- Can we detect and measure circuit formation before generalization emerges?
- How do biological brains organize computation across different cortical timescales?
- What makes a neural network circuit actually interpretable to humans?
- How do cortical columns implement local inference over memory cycles?
- Can fractured entangled representations hide undetected by standard analysis methods?
- Does the linear representation hypothesis reflect networks or reflect our analysis tools?
- How do knowledge and reasoning circuits interfere in the same neural network?
- How do neural networks decompose complex tasks into modular subnetworks?
- What inductive biases help networks segregate entities from raw inputs?
- What are fractured entangled representations in neural networks?
- How do sparse circuits compare to the modular subnetworks that emerge naturally?
- How does joint backpropagation differ from training separate ensemble models?
- How do gradients flowing through both branches simultaneously reshape each component's role?
- Can sub-task handlers be swapped between neural and symbolic systems?
- How does mechanistic interpretability complement learning mechanics in explaining deep learning?
- How do neural networks decompose tasks into modular subnetworks that transfer?
- Which hyperparameter theories best explain universal behaviors across neural networks?
- What solvable idealized settings reveal fundamental phenomena in realistic deep learning?
- How do classical mechanics and statistical mechanics provide methodological templates for learning theory?
- Can we predict which tasks will decompose into modular subnetworks?
- Why does gradient descent discover compositional structure without explicit pressure?
- How can neural networks be interpretable by design rather than post-hoc?
- What makes a feature abstract versus concrete in neural network activations?
- How does representational density emerge from training data familiarity?
- Why does knowledge storage separate from reasoning circuits in neural networks?
- How do concept vectors in neural networks relate to human-language chess explanations?
- Why did neuron interpretability jump from near-zero to over ninety percent?
- Does recursive generalization explain why neurons develop single interpretable functions?
- Why do some neurons detect multiple unrelated concepts in early models?
- Is the superposition hypothesis necessary to explain neural network behavior?
- Do dense models already achieve high interpretability without sparse training?
- Why did early neural networks need massive computing power for simple image tasks?
- Does information stored in neural networks necessarily influence generation decisions?
- Can steering vectors prove that representations are genuinely organized?
- Can neural networks learn that A implies B in reverse?
- Does causal intervention alone explain how neural mechanisms implement representations?
- How does nesting optimization levels improve on traditional network depth?
- What distinguishes hierarchical dual-recurrence from flat parameter-sharing recurrence?
- Can latent recurrence overcome the trainability costs of depth?
- Why does recursion on latent state drive generalization better than hierarchy?
- Can a single recursive network replace hierarchical dual-network architectures?
- What makes recurrent depth enable compositional generalization across tasks?
- Can a two-layer network outgeneralize billion-parameter models through recursion alone?
- Why does recursion on latent states improve generalization more than scale?
- What tasks does recurrent depth solve that feedforward models cannot?
- Can recurrent blocks learn genuinely novel computation beyond repetition?
- How do functional features differ from representational abstract features?
- What makes linear decodability a reliable signal of compositionality?
- What test distinguishes genuine compositionality from fractured feature presence?
- What role does a model's representational structure play in learning?
- Can geometric structure in representations exist without supporting functional mechanisms?
- How do ablation studies reveal function without representational characterization?
- What happens to representational structure during model pretraining phases?
- How do models develop dense representations for familiar training data?
- Can representation analysis methods detect complex features models compute with?
- What prevents representation collapse in latent-prediction world models like JEPA?
- How do semantic features in representations become steerable task-specific directions?
- How does requential coding measure true simplicity without parameter count inflation?
- What makes a new representational primitive valuable enough to justify its representational cost?
- What makes multimodal conditioning effective when features are decomposed to the right granularity?
- How do sparse networks trade capability for human-understandable circuits?
- Can finetuning sparse subnetworks alone match full parameter finetuning results?
- Why do singular value experts compose better than low-rank adapter subspaces?
- Why does weight sparsity reduce superposition and force disentangled representations?
- Can sparse approximations reveal interpretable structure hidden in existing dense models?
- How can interpretability methods account for shifting representational density across task conditions?
- Does sparsity enforce compositional structure or merely amplify existing modularity?
- Do KANs maintain their advantages in deep architectures and large-scale training?
- Can spiking sparsity replace weight quantization as a primary efficiency lever?
- Can dense models match specialized architectures by mixing data better?
- Can architectural changes reduce polysemanticity without explicit sparsity training?
- How do multimodal AI architectures compare to human brain export pathways?
- What does a human-parseable framework for deep learning look like?
- Are detection and identification of injections truly separable in neural circuits?
- Can neural grafts reliably reveal hidden capabilities in AI models?
- What performance trade-offs emerge when composing multiple independently trained model capabilities?
- Can expert vectors learned offline transfer across multiple model architectures?
- How much does pretraining quality affect the modularity of fine-tuned models?
- Can LLMs reliably generate novel working architectures without structured representations?
- How does neuro-symbolic design differ from pure LLM reasoning?
- How much do structural inductive biases matter compared to training data volume?
- Can training order and structure shape what networks retain and learn?
- What separates knowledge from reasoning in neural network layers?
- Can unfilled cells in the periodic table represent undiscovered argument schemes?
- Why do higher network layers capture procedural knowledge but lower layers store facts?
- Can we predict out-of-distribution generalization without access to downstream tasks?
- Why should deep learning theory prioritize average-case over worst-case analysis?
- Why does decomposition ability transfer across domains but solving ability does not?
- Can modular expert decomposition extend beyond time into other causal dimensions?
- Does hierarchical abstraction help equally across different manipulation tasks?
Related concepts in this collection 3
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Concept map
Do neural networks naturally learn modular compo…
Can sparse weight training make neural networks in…
Do base models already contain hidden reasoning ab…
Can neural networks learn compositional skills wit…
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
your link
semantically near
linked from elsewhere
-
Can sparse weight training make neural networks interpretable by design?
Explores whether constraining most model weights to zero during training produces human-understandable circuits and disentangled representations, rather than attempting to reverse-engineer dense models after training.
sparsity amplifies the compositional decomposition that standard training already partially produces
-
Do base models already contain hidden reasoning ability?
Explores whether reasoning capability emerges during pre-training as a latent feature rather than being created by post-training methods like reinforcement learning or fine-tuning.
pretraining-induced modularity is part of the "latent capability" that minimal signals can activate
-
Can neural networks learn compositional skills without symbolic mechanisms?
Do neural networks need explicit symbolic architecture to compose learned concepts, or can scaling alone enable compositional generalization? This asks whether compositionality is an architectural feature or an emergent property of scale.
complementary evidence: scaling enables compositionality in behavior; pruning reveals it in structure
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Break It Down: Evidence for Structural Compositionality in Neural Networks
- Scaling can lead to compositional generalization
- From Frege to chatGPT: Compositionality in language, cognition, and deep neural networks
- The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search
- Faith and Fate: Limits of Transformers on Compositionality
- Towards Monosemanticity: Decomposing Language Models With Dictionary Learning
- How do Transformers Learn Implicit Reasoning?
- Hierarchical Reasoning Model
Original note title
neural networks decompose compositional tasks into modular subnetworks without explicit symbolic mechanisms — pretraining encourages this