SYNTHESIS NOTE
Do neural networks naturally learn modular compositional structure?
Explores whether neural networks decompose compositional tasks into distinct subroutines without explicit symbolic design. This challenges the longstanding view that neural networks are fundamentally non-compositional.
Structural compositionality is the extent to which neural networks break down compositional tasks into subroutines and implement them in modular subnetworks. The alternative: matching inputs to learned templates without task decomposition.
The evidence supports compositionality. Using model pruning to isolate subnetworks:
- Subnetworks that implement one subroutine can be identified
- Ablating a subnetwork harms its corresponding subroutine while leaving others largely intact
- This holds across multiple architectures (CNNs, transformers), tasks (vision, language), and scales
The pretraining effect: models initialized with pretrained weights more reliably produce modular subnetworks than randomly initialized models. Self-supervised pretraining appears to create internal structure that is more amenable to compositional decomposition. This suggests that the representations learned during pretraining have a modular quality that fine-tuning can exploit.
This provides empirical support against the longstanding objection that neural networks are fundamentally non-compositional. The finding: "some simple pseudo-symbolic computations might be learned directly from data using standard gradient-based optimization techniques." Explicit symbolic mechanisms may be unnecessary — gradient-based optimization discovers compositional structure when the task demands it and pretraining provides a good initialization.
The result is not perfect: "most do not exhibit perfect task decomposition." Compositionality is partial and graded, not all-or-nothing. Some architecture-task combinations show stronger structural compositionality than others.
This connects to the weight-sparsity finding: Can sparse weight training make neural networks interpretable by design? shows that enforcing sparsity produces clean decomposition. The structural compositionality paper shows that decomposition also emerges naturally, albeit imperfectly, from standard training. Sparsity amplifies a tendency that already exists.
Inquiring lines that read this note 127
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
What determines success in training models on multiple tasks?- Can granular sub-task training for function calling improve both open and proprietary models?
- Does task superposition explain how models learn from multiple in-context trajectories?
- What performance trade-offs emerge when composing multiple independently trained model capabilities?
- How do neural networks decompose complex tasks into modular subnetworks?
- Does training on granular tasks beat training on the full function calling problem?
- How does joint backpropagation differ from training separate ensemble models?
- How do gradients flowing through both branches simultaneously reshape each component's role?
- Can sub-task handlers be swapped between neural and symbolic systems?
- How do neural networks decompose tasks into modular subnetworks that transfer?
- Can we predict which tasks will decompose into modular subnetworks?
- Could superposed decoding algorithms maintain multi-task representation during generation?
- Why do singular value experts compose better than low-rank adapter subspaces?
- Do transformers learn generalizable algorithms or instance-based patterns?
- Can symbolic mechanisms improve transformer compositional abilities?
- Can neural networks implement genuine algorithms or only statistical pattern matching?
- Could graph neural networks fundamentally outperform transformers on structured reasoning?
- Why do standard transformers fail to encode recursive structure in their hidden states?
- How does layer removal affect transformers compared to ResNets?
- How do induction heads learn to overwrite computational representations?
- Can spline-based activations replace MLPs in transformer architectures?
- What tasks does recurrent depth solve that feedforward models cannot?
- Can recurrent blocks learn genuinely novel computation beyond repetition?
- Can transformers abstract relational structure without explicit symbolic machinery?
- Can neural networks represent symbolic structures without explicit mechanisms?
- Does information stored in neural networks necessarily influence generation decisions?
- How do functional features differ from representational abstract features?
- Could probing methods miss computationally important features in neural networks?
- What makes linear decodability a reliable signal of compositionality?
- Can fractured representations explain why models fail at systematic generalization?
- Can we detect and measure circuit formation before generalization emerges?
- Are detection and identification of injections truly separable in neural circuits?
- What makes a neural network circuit actually interpretable to humans?
- Can fractured entangled representations hide undetected by standard analysis methods?
- Does the linear representation hypothesis reflect networks or reflect our analysis tools?
- What role does a model's representational structure play in learning?
- What inductive biases help networks segregate entities from raw inputs?
- What are fractured entangled representations in neural networks?
- How do sparse circuits compare to the modular subnetworks that emerge naturally?
- Can sparse approximations reveal interpretable structure hidden in existing dense models?
- How can interpretability methods account for shifting representational density across task conditions?
- Does causal intervention alone explain how neural mechanisms implement representations?
- Can geometric structure in representations exist without supporting functional mechanisms?
- How does mechanistic interpretability complement learning mechanics in explaining deep learning?
- Which hyperparameter theories best explain universal behaviors across neural networks?
- What solvable idealized settings reveal fundamental phenomena in realistic deep learning?
- How do classical mechanics and statistical mechanics provide methodological templates for learning theory?
- How do ablation studies reveal function without representational characterization?
- Can representation analysis methods detect complex features models compute with?
- How can neural networks be interpretable by design rather than post-hoc?
- What makes a feature abstract versus concrete in neural network activations?
- What prevents representation collapse in latent-prediction world models like JEPA?
- How does nesting optimization levels improve on traditional network depth?
- Do substitute networks converge differently than complement networks?
- What distinguishes hierarchical dual-recurrence from flat parameter-sharing recurrence?
- Can unfilled cells in the periodic table represent undiscovered argument schemes?
- Why do human-designed neural architectures eventually get replaced by learned ones?
- How do biological brains organize computation across different cortical timescales?
- Why are polysemantic features concentrated in early neural network layers?
- Why do text-to-image models fail at composing multiple concepts together?
- Does scaling model size solve compositional generalization problems?
- Does compositional generalization emerge suddenly or improve smoothly with scale?
- Does scaling data automatically produce compositional reasoning or just better feature encoding?
- What test distinguishes genuine compositionality from fractured feature presence?
- What makes recursive structure different from other forms of compositional generalization?
- Can scaling alone create compositional generalization without explicit binding mechanisms?
- Can granular function calling tasks learn composition from graph-sampled data?
- Can learned verifiers over token similarity replace dense compositional training?
- What role does query-level exposure play in enabling compositional generalization?
- Why does scaling data and model size improve compositional generalization?
- Does sparsity enforce compositional structure or merely amplify existing modularity?
- Why does gradient descent discover compositional structure without explicit pressure?
- What architectural alternatives can capture compositional structure beyond pooled cosine?
- What makes recurrent depth enable compositional generalization across tasks?
- How does scaling and training data enable compositional behavior without symbolic mechanisms?
- Where do neural networks still fail at compositional generalization despite scaling?
- Why does recursion on latent states improve generalization more than scale?
- Why does compositional reasoning fail to explain cross-domain transfer?
- What makes multimodal conditioning effective when features are decomposed to the right granularity?
- How do multimodal AI architectures compare to human brain export pathways?
- Can steering vectors prove that representations are genuinely organized?
- How do semantic features in representations become steerable task-specific directions?
- How do internal representations compare to human cognitive structures?
- Does directional knowledge failure indicate shallow pattern matching over deep representation?
- Can identical model performance mask fundamentally broken internal representations?
- Why should deep learning theory prioritize average-case over worst-case analysis?
- Do generic kernel-decay assumptions alone explain coarse-to-fine spectral ordering?
- How do sparse networks trade capability for human-understandable circuits?
- Why does weight sparsity reduce superposition and force disentangled representations?
- Can spiking sparsity replace weight quantization as a primary efficiency lever?
- How do cortical columns implement local inference over memory cycles?
- Can neural modules memorize surprising tokens as adaptive long-term memory?
- How does the hippocampus bind disparate elements without storing everything itself?
- Can LLMs reliably generate novel working architectures without structured representations?
- How does neuro-symbolic design differ from pure LLM reasoning?
- How much do structural inductive biases matter compared to training data volume?
- Can a two-layer network outgeneralize billion-parameter models through recursion alone?
- What separates knowledge from reasoning in neural network layers?
- How do knowledge and reasoning circuits interfere in the same neural network?
- Why do higher network layers capture procedural knowledge but lower layers store facts?
- Why does knowledge storage separate from reasoning circuits in neural networks?
- Can finetuning sparse subnetworks alone match full parameter finetuning results?
- Can expert vectors learned offline transfer across multiple model architectures?
- Can we predict out-of-distribution generalization without access to downstream tasks?
- How much does pretraining quality affect the modularity of fine-tuned models?
- What makes modernized N-gram embeddings composable with transformer architectures?
- Can spectral eigenvector ordering serve as a model-agnostic interpretability probe?
- Can single-vector embeddings capture non-commutative relationships like word order?
- How does representation-level reranking address residual gaps after decomposition?
- What happens to representational structure during model pretraining phases?
- How do models develop dense representations for familiar training data?
- Can latent recurrence overcome the trainability costs of depth?
- Do KANs maintain their advantages in deep architectures and large-scale training?
- How does representational density emerge from training data familiarity?
- Can training order and structure shape what networks retain and learn?
Related concepts in this collection 3
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Concept map
Do neural networks naturally learn modular compo…
Can sparse weight training make neural networks in…
Do base models already contain hidden reasoning ab…
Can neural networks learn compositional skills wit…
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
your link
semantically near
linked from elsewhere
-
Can sparse weight training make neural networks interpretable by design?
Explores whether constraining most model weights to zero during training produces human-understandable circuits and disentangled representations, rather than attempting to reverse-engineer dense models after training.
sparsity amplifies the compositional decomposition that standard training already partially produces
-
Do base models already contain hidden reasoning ability?
Explores whether reasoning capability emerges during pre-training as a latent feature rather than being created by post-training methods like reinforcement learning or fine-tuning.
pretraining-induced modularity is part of the "latent capability" that minimal signals can activate
-
Can neural networks learn compositional skills without symbolic mechanisms?
Do neural networks need explicit symbolic architecture to compose learned concepts, or can scaling alone enable compositional generalization? This asks whether compositionality is an architectural feature or an emergent property of scale.
complementary evidence: scaling enables compositionality in behavior; pruning reveals it in structure
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Break It Down: Evidence for Structural Compositionality in Neural Networks
- Scaling can lead to compositional generalization
- From Frege to chatGPT: Compositionality in language, cognition, and deep neural networks
- Faith and Fate: Limits of Transformers on Compositionality
- Towards Monosemanticity: Decomposing Language Models With Dictionary Learning
- How do Transformers Learn Implicit Reasoning?
- Hierarchical Reasoning Model
- Eliciting Reasoning in Language Models with Cognitive Tools
Original note title
neural networks decompose compositional tasks into modular subnetworks without explicit symbolic mechanisms — pretraining encourages this