TOPIC
Mobile and On-Device LLMs
A subject the collection covers, read through 4 synthesis notes.
View as
Does depth matter more than width for tiny language models?
Explores whether deep-and-thin architectures outperform wide-and-shallow ones at sub-billion scales, and why this might contradict larger-model scaling laws.
What actually limits language models on mobile phones? Does recomputing weights cost less than moving them on mobile? Can architecture choices improve inference efficiency without sacrificing accuracy? Can parallel architectures solve inherently sequential problems? How do language models learn to think like humans?
Mobile LLMs are constrained by hardware limits not quality preference Recomputing blocks costs less than moving weights on mobile devices Architectural variables enable 42% inference efficiency gains while improving accuracy Parallel architectures cannot solve inherently sequential problems Language models develop human-like cognitive biases and reasoning patterns
Why do negative item weights matter more than model depth? What structural constraints matter more than model depth for CF? Why does frame-activation matter more than word-by-word composition? Can input augmentation and rephrasing compensate for smaller model limitations? How do embedding dimension limits constrain what concept models can represent? Do modern architectures in NLP and vision rely on dot products intentionally? What compression explains why syntax fits in low-dimensional subspaces? How does nesting optimization levels improve on traditional network depth? Does architectural discovery follow an empirical scaling law like neural networks? Can adaptive compute allocation at sub-token granularity improve cross-lingual robustness? Why are polysemantic features concentrated in early neural network layers? Do larger models develop more abstract features than smaller ones?See all 111 inquiring lines on this note →
Does recomputing weights cost less than moving them on mobile?
Explores whether mobile hardware's memory bottleneck makes it cheaper to recompute transformer blocks than to fetch their weights twice, and whether this trades accuracy for efficiency.
Does depth matter more than width for tiny language models? What actually limits language models on mobile phones? Can architecture choices improve inference efficiency without sacrificing accuracy? Can ternary weights match full precision model performance? How do language models learn to think like humans?
Depth beats width for sub-billion parameter language models Mobile LLMs are constrained by hardware limits not quality preference Architectural variables enable 42% inference efficiency gains while improving accuracy Ternary 1.58-bit weights match full precision performance identically Language models develop human-like cognitive biases and reasoning patterns
Can offline context optimization reduce test-time latency like sleep-time compute? What constraints force mobile deployments to operate in the sub-billion parameter regime? What decomposition level minimizes both error rate and computational cost in practice? What mobile hardware constraints force the sub-billion parameter regime? How do conditional scaling laws incorporate hardware into architecture choices? How does adjacent layer sharing differ from non-adjacent weight reuse? Why would compute-replacement cost determine wages instead of productivity? Can layer-wise KV caches enable truly lossless information transfer? How does layer removal affect transformers compared to ResNets? Why does recomputing weights cost less than moving them on phones? Could deploying GPT-4 for everyone require 100 million specialized chips? What computational costs does closed-loop memory refinement introduce?See all 23 inquiring lines on this note →
What actually limits language models on mobile phones?
Is the shift toward smaller LLMs driven by quality trade-offs, or by hard physical constraints on device memory and battery life? This note examines whether sub-billion models are a practical necessity rather than a compromise.
Does depth matter more than width for tiny language models? Does recomputing weights cost less than moving them on mobile? Can small language models handle most agent tasks? Can architecture choices improve inference efficiency without sacrificing accuracy? Can inference compute replace scaling up model size?
Depth beats width for sub-billion parameter language models Recomputing blocks costs less than moving weights on mobile devices Small language models suffice for most agentic subtasks cost-effectively Architectural variables enable 42% inference efficiency gains while improving accuracy Smaller models with more inference compute match larger ones on hard prompts
Can routing enable heterogeneous SLM-first architectures at scale? What constraints force mobile deployments to operate in the sub-billion parameter regime? Does the optimal model size depend on what capabilities you actually need? Can smaller models actually perform well on specific downstream tasks? How do input length constraints reshape personalization system design choices? What mobile hardware constraints force the sub-billion parameter regime? How should tiny language models be architected differently than large ones? Why does recomputing weights cost less than moving them on phones? Which architectural choices matter most when a model must fit one billion parameters?
Can ternary weights match full precision model performance?
Can models trained natively with only three weight values (−1, 0, 1) achieve the same perplexity and task performance as standard full-precision models? This matters because ternary weights could dramatically reduce computational and energy costs.
Can spiking neurons make transformers efficient on any hardware? Can architecture choices improve inference efficiency without sacrificing accuracy? What actually limits language models on mobile phones? Does recomputing weights cost less than moving them on mobile? Do hidden massive activations act as attention bias terms?
Spiking linear attention converts transformers into efficient models with minimal retraining Architectural variables enable 42% inference efficiency gains while improving accuracy Mobile LLMs are constrained by hardware limits not quality preference Recomputing blocks costs less than moving weights on mobile devices Massive activations function as implicit attention bias terms in LLMs