SYNTHESIS NOTE
Does depth matter more than width for tiny language models?
Explores whether deep-and-thin architectures outperform wide-and-shallow ones at sub-billion scales, and why this might contradict larger-model scaling laws.
Kaplan et al.'s scaling laws establish a roughly balanced relationship between model depth and width as parameters scale, with width growth often dominating at typical model sizes. MobileLLM demonstrates that this guidance breaks at the sub-billion-parameter scale relevant for on-device deployment. A deep-and-thin model structure outperforms balanced or wide-and-shallow alternatives, producing 2.7 percent and 4.3 percent accuracy boosts over preceding 125M and 350M state-of-the-art models respectively. The reason offered is that depth captures abstract concepts — composing simpler features into hierarchical representations through more layers — and at small scale the model has fewer raw parameters to spend, so making each one work harder through compositional depth pays back more than spreading them across wider layers.
This matters because it shows that scaling laws are regime-dependent rather than universal. The Kaplan results were derived from larger models where width and depth are both abundant; at the small scale where mobile deployment lives, the trade-offs reverse. The implication is that the architectural recipe for on-device LLMs is genuinely different from the recipe for cloud-scale LLMs — not just smaller, but structurally different. Can architecture choices improve inference efficiency without sacrificing accuracy? makes the same point at the inference-economics layer: vanilla scaling laws say nothing about deployment regimes.
The deeper lesson is methodological: scaling laws should always be qualified by the regime in which they were derived, and recommendations for sub-billion-parameter design should not be extrapolated downward from billion-plus-parameter studies. The right architecture for a 350M parameter model is not a scaled-down version of a 70B parameter model; it is a deep-and-thin model derived from the constraints of the small-scale regime. Can parallel architectures solve inherently sequential problems? gives a complementary reason to favor depth — some computations require sequential composition that width cannot supply at any scale.
Inquiring lines that read this note 111
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
What structural factors drive popularity bias in recommendation systems?- Why do negative item weights matter more than model depth?
- Can lower embedding dimensions alone solve the diversity problem without attention mechanisms?
- What sparse high-rank patterns does the deep tower fail to capture?
- What structural constraints matter more than model depth for CF?
- How do embedding dimension limits constrain what concept models can represent?
- How do sub-token and architecture-level compute optimization strategies compare?
- Why do power-law distributions make standard ML infrastructure assumptions fail?
- Why does depth outperform width for sub-billion parameter models?
- What mobile hardware constraints force the sub-billion parameter regime?
- How does the Ladder of Scales approach reduce search costs across model sizes?
- Can depth scaling and breadth scaling unlock independent capability axes?
- How much do structural inductive biases matter compared to training data volume?
- Do small models show different parameter efficiency patterns than large models?
- How should tiny language models be architected differently than large ones?
- Which architectural choices matter most when a model must fit one billion parameters?
- What makes a small surgical wide component sufficient with a capable deep model?
- Can width-scaling replace depth-scaling on inherently sequential problems?
- Can a two-layer network outgeneralize billion-parameter models through recursion alone?
- Why does reused computation outperform adding new model depth?
- Do modern architectures in NLP and vision rely on dot products intentionally?
- Does bidirectional attention improve language models as universal encoders?
- What compression explains why syntax fits in low-dimensional subspaces?
- Why does adjusted compression performance degrade as models scale larger?
- Can model compression size predict generalization better than parameter count?
- Why do parameter-based compressors fail to measure true model simplicity?
- Does architectural discovery follow an empirical scaling law like neural networks?
- Why do scaling laws fail to predict optimal architectures at small parameter counts?
- How do conditional scaling laws incorporate hardware into architecture choices?
- What scaling laws govern autonomous architecture discovery in AI systems?
- What are the scaling law differences between vision and language learning?
- What power-law scaling patterns emerge when consistency models are trained at scale?
- Can adaptive compute allocation at sub-token granularity improve cross-lingual robustness?
- Can smaller models actually perform well on specific downstream tasks?
- Can smaller specialist models outperform large generalist models on domain tasks?
- Why does exploration quality matter more than learner network depth?
- Why are polysemantic features concentrated in early neural network layers?
- Do larger models develop more abstract features than smaller ones?
- Does scaling model size solve compositional generalization problems?
- Does compositional generalization emerge suddenly or improve smoothly with scale?
- Can scaling alone create compositional generalization without explicit binding mechanisms?
- Why does scaling data and model size improve compositional generalization?
- Does sparsity enforce compositional structure or merely amplify existing modularity?
- What architectural alternatives can capture compositional structure beyond pooled cosine?
- What makes recurrent depth enable compositional generalization across tasks?
- Why does recursion on latent states improve generalization more than scale?
- Can recursion alone drive generalization better than model scale?
- Does the model learn depth-wise drift as an explicit strategy?
- How do models develop dense representations for familiar training data?
- Can latent recurrence overcome the trainability costs of depth?
- Why should scaling laws be understood as properties of data distribution rather than training in general?
- Do KANs maintain their advantages in deep architectures and large-scale training?
- How does inference compute substitution affect the training parameter scaling trade-off?
- Why does architecture matter more than training compute for inference efficiency?
- Does flexible inference-time compute scaling through looping improve efficiency further?
- Can sequential computation through depth solve problems that parallel width cannot?
- How do parallel sampling and sequential depth compare as scaling dimensions?
- Can neural networks implement genuine algorithms or only statistical pattern matching?
- How does adjacent layer sharing differ from non-adjacent weight reuse?
- Can transformers reason beyond fixed architectural depth limits?
- Can bounded-depth transformers solve inherently sequential problems?
- What tasks does recurrent depth solve that feedforward models cannot?
- Can looped architectures achieve reasoning abilities that fixed-depth models cannot?
- How does hierarchical recurrence compare to selective layer looping for computational depth?
- How do you measure the depth of political representation inside a language model?
- What inductive biases help networks segregate entities from raw inputs?
- Which hyperparameter theories best explain universal behaviors across neural networks?
- What solvable idealized settings reveal fundamental phenomena in realistic deep learning?
- Why do large language models still have systematic blind spots with complex structures?
- Why do language models plateau at constraint satisfaction regardless of scale?
- Why do naive pruning and quantization destroy LLM performance so easily?
- Which linguistic abilities are learnable from human-sized data exposure?
- Do language models encode deep syntactic structure or only surface-level patterns?
- Can encoder models match human conceptual structure better than larger language models?
- Why do smaller models favor code formats while larger models prefer natural language?
- How do corpus statistics shape the abstraction hierarchy in language model representations?
- Do pretrained language models carry reusable computational scaffolding for length handling?
- Why do vision and language have different optimal scaling curves?
- What geometric structure do language models actually use during inference?
- How do parameter scaling and latent vectors interact in language models?
- Why do language models reproduce human EPA structure despite different architecture?
- Does directional knowledge failure indicate shallow pattern matching over deep representation?
- What empirical evidence supports the Learning Law on real language models?
- Why does weight sparsity reduce superposition and force disentangled representations?
- Why do hybrid memory and compute sparsity outperform pure parameter scaling?
- How should benchmark design account for task-dependent sparsity tolerance differences?
- How does modality-specific sparsity enable capacity flexibility that dense models cannot provide?
- What makes looped latent computation more efficient than scaling attention capacity?
- Can spiking sparsity replace weight quantization as a primary efficiency lever?
- Can attention linearity achieve similar efficiency gains as weight quantization?
- How should GPU execution paths and training objectives co-design sparsity?
- What makes modernized N-gram embeddings composable with transformer architectures?
- Can small transformers trained on similarity maps replace dense retrievers entirely?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Concept map
Does depth matter more than width for tiny langu…
What actually limits language models on mobile pho…
Does recomputing weights cost less than moving the…
Can architecture choices improve inference efficie…
Can parallel architectures solve inherently sequen…
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
your link
semantically near
linked from elsewhere
-
What actually limits language models on mobile phones?
Is the shift toward smaller LLMs driven by quality trade-offs, or by hard physical constraints on device memory and battery life? This note examines whether sub-billion models are a practical necessity rather than a compromise.
extends: same MobileLLM source; this note answers WHY sub-billion is the regime, depth-vs-width answers HOW to design within it
-
Does recomputing weights cost less than moving them on mobile?
Explores whether mobile hardware's memory bottleneck makes it cheaper to recompute transformer blocks than to fetch their weights twice, and whether this trades accuracy for efficiency.
extends: same MobileLLM paper; depth wins partly because depth-with-shared-weights can be deeper than depth-with-distinct-weights at fixed parameter count; the two design moves compound
-
Can architecture choices improve inference efficiency without sacrificing accuracy?
Standard scaling laws optimize training efficiency but ignore inference cost. This explores whether architectural variables like hidden size and attention configuration can unlock inference gains without trading off model accuracy under fixed training budgets.
extends: both reject regime-blind scaling laws; this note shows depth-width trade-offs flip in the small regime; conditional scaling laws formalize how architecture variables modulate the law
-
Can parallel architectures solve inherently sequential problems?
Complexity theory suggests some problems like reasoning and planning are fundamentally sequential. Can parallel architectures like Transformers overcome this limitation, or do we need fundamentally different computational approaches?
extends: gives a theoretical reason to prefer depth (serial composition) over width (parallel breadth) for capability-bounded models
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases
- Hierarchical Reasoning Model
- Scaling Laws for Neural Language Models
- Nested Learning: The Illusion of Deep Learning Architectures
- Bigger is not always better: The importance of human-scale language modeling for psycholinguistics
- Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models
- Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs
- Large Language Diffusion Models
Original note title
depth beats width for sub-billion parameter LLMs — contradicting Kaplan scaling laws because deep-and-thin captures abstract concepts better at small scale