Line of inquiry
Inquiring lines›How do training and design choices…›What determines whether training i…›this line of inquiry
How do neural networks achieve compositional generalization at scale?
A broader line of inquiry — a family of 68 specific questions the research asks around this. Follow one into its inquiring-line page, or move sideways to a related line below.
Questions in this line of inquiry 68
Specific inquiring lines the field asks around this — ordered from the most general framing down to the most specific angle.
- How does scaling and training data enable compositional behavior without symbolic mechanisms?
- Does the linear representation hypothesis reflect networks or reflect our analysis tools?
- Can identical model performance mask fundamentally broken internal representations?
- Can scaling alone create compositional generalization without explicit binding mechanisms?
- Where do neural networks still fail at compositional generalization despite scaling?
- Why does gradient descent discover compositional structure without explicit pressure?
- Does latent density emerge during pretraining from training data familiarity?
- Can fractured representations explain why models fail at systematic generalization?
- How does representational density emerge from training data familiarity?
- Can neural networks represent symbolic structures without explicit mechanisms?
- What inductive bias would force models to learn Newtonian mechanics instead of shortcuts?
- What prevents LLM representations from causally influencing generation outputs?
- Can geometric structure in representations exist without supporting functional mechanisms?
- Why do internal representations differ when external performance matches?
- Can steering vectors prove that representations are genuinely organized?
- How do latents at the same hierarchy level become more correlated than tokens?
- Do larger models develop more abstract features than smaller ones?
- Does representational density emerge from training data exposure during pretraining?
- Can spectral eigenvector ordering serve as a model-agnostic interpretability probe?
- Can fractured entangled representations hide undetected by standard analysis methods?
- What are fractured entangled representations in neural networks?
- What role does a model's representational structure play in learning?
- How much do structural inductive biases matter compared to training data volume?
- Can representation engineering cleanly isolate single features in entangled semantic space?
- Does architectural discovery follow an empirical scaling law like neural networks?
- Can representation analysis methods detect complex features models compute with?
- What makes a feature abstract versus concrete in neural network activations?
- How should we rethink the symbolism versus connectionism debate in light of LLMs?
- Can neural networks implement genuine algorithms or only statistical pattern matching?
- Do KANs maintain their advantages in deep architectures and large-scale training?
- What inductive biases help networks segregate entities from raw inputs?
- Why is latent-level prediction more sample-efficient than token-level prediction?
- What prevents representation collapse in latent-prediction world models like JEPA?
- Why do human-designed neural architectures eventually get replaced by learned ones?
- What role does query-level exposure play in enabling compositional generalization?
- Do SAE features explain decision-making in LLM agents better than probes?
- Does the same spectral signature appear across different embedding models?
- Does information stored in neural networks necessarily influence generation decisions?
- How can neural networks be interpretable by design rather than post-hoc?
- What makes linear decodability a reliable signal of compositionality?
- What happens to representational structure during model pretraining phases?
- Can generative reconstruction preserve latent manifold structure better than geometric compression?
- What non-linear patterns do autoencoders discover that matrix factorization misses?
- Could probing methods miss computationally important features in neural networks?
- Can neural networks learn that A implies B in reverse?
- Can universal function approximators be expensive to learn in practice?
- Does compositional generalization emerge suddenly or improve smoothly with scale?
- How do neural networks decompose complex tasks into modular subnetworks?
- Why does scaling data and model size improve compositional generalization?
- How do encode-decode contractive biases create stable attractors in latent space?
- What solvable idealized settings reveal fundamental phenomena in realistic deep learning?
- Can curvature measurements predict task difficulty without behavioral labels?
- What scaling laws govern the compute efficiency of latent prediction versus token prediction?
- What makes regularization an implicit factor in embedding geometry?
- Do feature extraction methods systematically miss computationally important complex features?
- Can autoencoders act as associative memory systems like Hopfield networks?
- What limits the extrapolation of learned operations like rotation and reflection?
- How do sparse circuits compare to the modular subnetworks that emerge naturally?
- What happens when a single loss function conflates representation learning with decision-making?
- How do classical mechanics and statistical mechanics provide methodological templates for learning theory?
- What non-parametric methods could replace latent factors for inductive learning?
- What physical structure does a Gaussian-regularized latent space actually encode?
- What makes a self-supervised pruning metric work without labels at scale?
- Do generic kernel-decay assumptions alone explain coarse-to-fine spectral ordering?
- How do neural networks extend contextual bandits beyond linear reward assumptions?
- Which hyperparameter theories best explain universal behaviors across neural networks?
- Can other posterior approximation schemes match variational inference performance?
- Why do different brain and AI systems appear similar when compared via RSA?