Does depth matter more than width for tiny language models?
Explores whether deep-and-thin architectures outperform wide-and-shallow ones at sub-billion scales, and why this might contradict larger-model scaling laws.
What actually limits language models on mobile phones? Does recomputing weights cost less than moving them on mobile? Can architecture choices improve inference efficiency without sacrificing accuracy? Can parallel architectures solve inherently sequential problems? How do language models learn to think like humans?
Mobile LLMs are constrained by hardware limits not quality preference Recomputing blocks costs less than moving weights on mobile devices Architectural variables enable 42% inference efficiency gains while improving accuracy Parallel architectures cannot solve inherently sequential problems Language models develop human-like cognitive biases and reasoning patterns
Why do negative item weights matter more than model depth? What structural constraints matter more than model depth for CF? Why does frame-activation matter more than word-by-word composition? Can input augmentation and rephrasing compensate for smaller model limitations? How do embedding dimension limits constrain what concept models can represent? Do modern architectures in NLP and vision rely on dot products intentionally? What compression explains why syntax fits in low-dimensional subspaces? How does nesting optimization levels improve on traditional network depth? Does architectural discovery follow an empirical scaling law like neural networks? Can adaptive compute allocation at sub-token granularity improve cross-lingual robustness? Why are polysemantic features concentrated in early neural network layers? Do larger models develop more abstract features than smaller ones?See all 97 inquiring lines on this note →