SYNTHESIS NOTE
Topics›Mobile›this note

Does depth matter more than width for tiny language models?

Explores whether deep-and-thin architectures outperform wide-and-shallow ones at sub-billion scales, and why this might contradict larger-model scaling laws.

Synthesis note · 2026-05-03 · sourced from Mobile

Kaplan et al.'s scaling laws establish a roughly balanced relationship between model depth and width as parameters scale, with width growth often dominating at typical model sizes. MobileLLM demonstrates that this guidance breaks at the sub-billion-parameter scale relevant for on-device deployment. A deep-and-thin model structure outperforms balanced or wide-and-shallow alternatives, producing 2.7 percent and 4.3 percent accuracy boosts over preceding 125M and 350M state-of-the-art models respectively. The reason offered is that depth captures abstract concepts — composing simpler features into hierarchical representations through more layers — and at small scale the model has fewer raw parameters to spend, so making each one work harder through compositional depth pays back more than spreading them across wider layers.

This matters because it shows that scaling laws are regime-dependent rather than universal. The Kaplan results were derived from larger models where width and depth are both abundant; at the small scale where mobile deployment lives, the trade-offs reverse. The implication is that the architectural recipe for on-device LLMs is genuinely different from the recipe for cloud-scale LLMs — not just smaller, but structurally different. Can architecture choices improve inference efficiency without sacrificing accuracy? makes the same point at the inference-economics layer: vanilla scaling laws say nothing about deployment regimes.

The deeper lesson is methodological: scaling laws should always be qualified by the regime in which they were derived, and recommendations for sub-billion-parameter design should not be extrapolated downward from billion-plus-parameter studies. The right architecture for a 350M parameter model is not a scaled-down version of a 70B parameter model; it is a deep-and-thin model derived from the constraints of the small-scale regime. Can parallel architectures solve inherently sequential problems? gives a complementary reason to favor depth — some computations require sequential composition that width cannot supply at any scale.

Inquiring lines that read this note 133

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

Can structural constraints replace embedding depth in collaborative filtering? How does explicit structure improve LLM reasoning on complex tasks? What distinguishes genuine understanding from correct linguistic performance? Does prompting inject new knowledge into models or activate existing knowledge? How do mechanistic features and representations drive model behavior? How do transformer circuits and attention mechanisms enable or limit reasoning? Does learning articulatory physics explain speech communication superiority? Does architectural recurrence enable computational capabilities that fixed-depth models cannot achieve? What determines how neural networks organize their internal knowledge representations? How can parameter sparsity improve interpretability while maintaining model capability? Can scaling alone produce compositional generalization in neural networks? What training data structures and signals produce robust learning across distributions? How does scaling test-time compute compare to increasing model size? When do additional thinking tokens stop improving reasoning performance? Why does separating planning from execution enhance reasoning performance? How does model scale affect fine-tuning and downstream task performance? What explains the gap between LLM semantic understanding and production? Do language models learn underlying linguistic structure or merely surface patterns? Do language models truly understand their encoded knowledge or merely retrieve patterns? Does training data format shape reasoning more than domain content? Can AI systems autonomously pursue unstated objectives or circumvent design constraints? Do language models genuinely understand or primarily match statistical patterns? Does parallel reasoning outperform sequential approaches in reasoning efficiency? When does routing outperform scaling as a performance solution? What fundamental limitations prevent embeddings from solving task-relevant retrieval? How do benchmark scores relate to real-world task completion? Do AI explanations represent actual model reasoning or constructed post-hoc narratives? Should GUI agents use structured UI perception instead of raw pixels? Does semantic diversity improve reasoning quality in reinforcement learning? How should test-time compute be allocated adaptively across task difficulties? Can we reliably detect when models engage in reward hacking? When do multi-agent systems outperform single agents? How do diffusion language models compare to autoregressive architectures? Do standard benchmarks reveal AI capability in real-world contexts? How can language models effectively manage long-context memory without quality degradation?

Related concepts in this collection 4

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
13 direct connections · 107 in 2-hop network ·medium cluster Open in graph ↗
Does depth matter more than width for tiny langu… What actually limits language models on mobile pho… Does recomputing weights cost less than moving the… Can architecture choices improve inference efficie… Can parallel architectures solve inherently sequen…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

Why do scaling laws fail to predict optimal architectures at small parameter counts?How does weight sharing compound the advantages of deeper model designs?Can sequential computation through depth solve problems that parallel width cannot?What constraints force mobile deployments to operate in the sub-billion parameter regime?
Original note title

depth beats width for sub-billion parameter LLMs — contradicting Kaplan scaling laws because deep-and-thin captures abstract concepts better at small scale