Why do singular value experts compose better than low-rank adapter subspaces?
This explores why tuning the singular values inside a model's existing weight matrices yields experts that mix cleanly at inference, where stacking low-rank adapters (LoRA) tends to interfere — and what the corpus suggests about the deeper reason.
This explores why singular-value experts compose so cleanly while low-rank adapters tend to step on each other. The direct evidence comes from Transformer² Can models dynamically activate expert skills at inference time?, which tunes *only* the singular values of a weight matrix rather than adding a new low-rank update on top of it. The distinction matters: a LoRA adapter injects a fresh subspace into the model, and two adapters trained separately can claim overlapping directions that collide when combined. Scaling singular values, by contrast, only re-weights directions the pretrained model already uses — it turns existing structure up or down instead of grafting on new structure. That's why the experts mix at inference without interference and do it with fewer parameters.
The corpus suggests the real lever here is *respecting structure the model already has* rather than imposing new capacity. Several notes point the same way. Neural networks, left to themselves, already decompose tasks into isolated modular subnetworks — ablate one and only its function breaks Do neural networks naturally learn modular compositional structure?. Pretraining makes this modularity more consistent and reliable. If composable structure is *already latent* in the weights, then a method that edits along those native axes (singular directions) inherits the modularity for free, while a method that adds arbitrary new subspaces has to hope they happen to align with it.
The opposite failure case is illuminating. A model can hold all the features it needs and still have fundamentally fractured internal organization that's invisible to accuracy metrics but fragile under perturbation Can models be smart without organized internal structure?. Bolting on low-rank subspaces is exactly the kind of move that can produce that fracture — it works on the benchmark but composes badly because the added directions aren't disentangled from the rest. Work on weight sparsity makes the converse point: when you *force* modularity (here through sparse weights), you get compact circuits that ablation studies confirm are clean and separable Can sparse weight training make neural networks interpretable by design?. Composition is easy when the pieces are genuinely disentangled, hard when they overlap.
There's a more radical version of the same intuition in the swarm-search work Can language models discover new expertise through collaborative weight search?: you can discover *new* composed experts just by moving particles through weight space — no gradients, no added parameters, even solving problems every starting expert failed. That only works if the weight space itself is the right coordinate system for combining skills. Singular-value tuning and weight-space search are two expressions of the same bet — that the model's existing geometry is where composition should happen, not in bolted-on side-channels.
The thread worth pulling, if you go deeper: the recurring lesson across this corpus is that **structural bias beats raw capacity.** A linear recommender with one well-chosen constraint beats deep models Can a linear model beat deep collaborative filtering?; forced sparsity buys interpretability; native modularity buys composability. Singular-value experts win for the same reason — not because they're more expressive, but because they're more honest about the structure already there.
Sources 6 notes
Transformer2 demonstrates that tuning only singular values within weight matrices produces composable expert vectors that dynamically mix at inference without interference, outperforming LoRA with fewer parameters and enabling continual specialization.
Pruning experiments reveal that neural networks implement compositional subroutines in isolated subnetworks, with ablations affecting only their corresponding function. Pretraining substantially increases the consistency and reliability of this modular structure across architectures and domains.
Models trained with SGD can contain all the linearly decodable features needed for a task while maintaining fundamentally broken internal organization. This makes them vulnerable to perturbation and distribution shift invisible to standard evaluation metrics.
Training transformers with sparse weights creates compact, human-interpretable circuits where neurons correspond to simple concepts with clear connections. Ablation studies confirm these circuits are necessary and sufficient for task performance, though scaling beyond tens of millions of parameters while maintaining interpretability remains unsolved.
PSO-inspired swarms of LLM particles moving through weight space discover composed experts with new capabilities—including answering questions all initial experts failed on—using only 200 validation examples and no gradient-based training.
ESLER, a single-layer linear autoencoder constrained so items cannot predict themselves, outperforms most deep CF models. The constraint forces prediction through item relationships, and negative weights encoding anti-affinity prove essential—structural bias matters more than model capacity.