SYNTHESIS NOTE
Topics›Mobile›this note

Does depth matter more than width for tiny language models?

Explores whether deep-and-thin architectures outperform wide-and-shallow ones at sub-billion scales, and why this might contradict larger-model scaling laws.

Synthesis note · 2026-05-03 · sourced from Mobile

Kaplan et al.'s scaling laws establish a roughly balanced relationship between model depth and width as parameters scale, with width growth often dominating at typical model sizes. MobileLLM demonstrates that this guidance breaks at the sub-billion-parameter scale relevant for on-device deployment. A deep-and-thin model structure outperforms balanced or wide-and-shallow alternatives, producing 2.7 percent and 4.3 percent accuracy boosts over preceding 125M and 350M state-of-the-art models respectively. The reason offered is that depth captures abstract concepts — composing simpler features into hierarchical representations through more layers — and at small scale the model has fewer raw parameters to spend, so making each one work harder through compositional depth pays back more than spreading them across wider layers.

This matters because it shows that scaling laws are regime-dependent rather than universal. The Kaplan results were derived from larger models where width and depth are both abundant; at the small scale where mobile deployment lives, the trade-offs reverse. The implication is that the architectural recipe for on-device LLMs is genuinely different from the recipe for cloud-scale LLMs — not just smaller, but structurally different. Can architecture choices improve inference efficiency without sacrificing accuracy? makes the same point at the inference-economics layer: vanilla scaling laws say nothing about deployment regimes.

The deeper lesson is methodological: scaling laws should always be qualified by the regime in which they were derived, and recommendations for sub-billion-parameter design should not be extrapolated downward from billion-plus-parameter studies. The right architecture for a 350M parameter model is not a scaled-down version of a 70B parameter model; it is a deep-and-thin model derived from the constraints of the small-scale regime. Can parallel architectures solve inherently sequential problems? gives a complementary reason to favor depth — some computations require sequential composition that width cannot supply at any scale.

Inquiring lines that read this note 133

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

How should recommendation systems balance competing signals and objectives? When do complex LLM tasks benefit from external verification or structured scaffolding? What distinguishes genuine linguistic understanding from mere pattern matching? Can prompting inject new knowledge into models or only activate existing capabilities? Why do embedding-based systems fail on task-relevant document retrieval? How do transformer attention mechanisms create position and sycophancy biases? How does compressing or distilling information affect model learning and generalization? Can recursive latent-state computation outperform model scale and hierarchy? Do small language models have fundamentally different scaling properties than large ones? Why do NLP benchmarks treat human disagreement as noise rather than signal? How do sparse and dense representations support compositionality in language models? How do training data properties affect model generalization and robustness? How should computational resources be optimally allocated during inference? When do thinking tokens stop improving reasoning performance? When does parallel reasoning outperform sequential reasoning under equivalent computational budgets? Can recurrence enable transformers to solve problems beyond fixed-depth limitations? How aligned are models' verbal explanations with their actual internal reasoning? Do language models learn genuine understanding or just surface patterns? Do language models rely on encoded knowledge or primarily on surface patterns? How do attention patterns affect model performance across varying context lengths? Why does fine-tuning for specific tasks degrade broader reasoning capability? Can AI systems achieve sustainable self-improvement without external human guidance? Can intelligent model routing compete with model scaling as an optimization lever? Do fine-tuning methods modify core model capabilities or just outputs? How should architects optimize KV cache memory and compute tradeoffs for inference? Does polished AI presentation mislead people about accuracy? Do LLMs lack causal understanding needed for accurate prediction? How do prompt injection attacks succeed or fail across defenses? When do multi-agent systems outperform single-agent approaches for complex task execution? How do diffusion language models enable capabilities autoregressive architectures cannot achieve? How do standardized benchmarks differ from measures of real-world AI capability?

Related concepts in this collection 4

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
13 direct connections · 110 in 2-hop network ·medium cluster Open in graph ↗

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Original note title

depth beats width for sub-billion parameter LLMs — contradicting Kaplan scaling laws because deep-and-thin captures abstract concepts better at small scale