Line of inquiry
Inquiring lines›How can we train models to be prop…›What determines the effectiveness…›this line of inquiry
Do small language models have fundamentally different scaling properties than large ones?
A broader line of inquiry — a family of 88 specific questions the research asks around this. Follow one into its inquiring-line page, or move sideways to a related line below.
Questions in this line of inquiry 88
Specific inquiring lines the field asks around this — ordered from the most general framing down to the most specific angle.
- Do smaller models optimized for benchmarks perform worse than their scores suggest in real tasks?
- Can smaller models actually perform well on specific downstream tasks?
- How should tiny language models be architected differently than large ones?
- Do smaller models match Chinchilla-optimal performance when trained much longer?
- How does model scale affect the crossover point between base and post-trained performance?
- Does fine-tuning a small model match fine-tuning a large one?
- Why do larger models reduce interference between rare and common tasks?
- Do small models show different parameter efficiency patterns than large models?
- Why do smaller and larger models converge on different output formats?
- Can externalized memory and skills replace model scaling?
- Can smaller specialist models outperform large generalist models on domain tasks?
- Why do small specialized models match frontier multimodal models on screen tasks?
- Can frontier-scale results be predicted from small-scale benchmark speedups?
- Can test-time compute fully replace scaling model parameters on hard problems?
- Do larger models develop more abstract features than smaller ones?
- Does small heterogeneous model architecture outperform large homogeneous pools economically?
- Why does the right structural prior matter more than raw model capacity?
- Can RL format selection explain performance gains attributed to algorithmic improvements?
- Why do scaling laws show capability saturation at specific thresholds?
- Does scaling model size solve compositional generalization problems?
- How do architectural changes affect inference efficiency independent of model size?
- Does memory granularity need to match the task domain or the model size?
- Do all models of equivalent quality see the same inference request volume?
- Can smaller language models match large model accuracy at lower total cost?
- What role does inductive bias play versus model capacity in practice?
- Why do scaling laws fail to predict optimal architectures at small parameter counts?
- What happens to the optimal model size as projected inference volume grows?
- Can task type alone predict which model size is sufficient?
- Does the optimal model size depend on what capabilities you actually need?
- How can smaller models help select useful data for larger models?
- What capabilities actually require massive scale versus specialized training regimes?
- Where exactly does the threshold between model sizes stop mitigation from working?
- How do larger models maintain more parallel tasks than smaller models?
- Can width-scaling replace depth-scaling on inherently sequential problems?
- What makes sparse models inefficient to train and deploy at scale?
- Why do small language models show faster on-premise payback than large ones?
- How do model size and document diversity interact in SDF override success?
- How small must the anchoring stream be to correct world model bias?
- What production constraints should determine paradigm selection?
- How does the hardware-aware scaling recipe enable looped models to win?
- Why does capability saturation and diversity saturation occur at different scales?
- Do different model sizes show different rates of optional field overfilling behavior?
- What makes a small surgical wide component sufficient with a capable deep model?
- How does model size within a family affect output distinguishability?
- How can expensive models efficiently support cheap models in production?
- Does architectural discovery follow an empirical scaling law like neural networks?
- Are architectural differences between models more detectable than size differences?
- Do leader-follower roles depend on specific scale gaps between paired models?
- Are cheap testbeds and skewed task distributions linked by design necessity?
- Are larger models and search access substitutes for factual accuracy?
- Can a complexity-predictor be meaningful if models are redundant?
- Can architectural changes reduce representational inequality in unified generators?
- How do task difficulty and skill type interact in model performance?
- What is minimum viable scale for deploying an AI system?
- Does the O(1)-bit signal theory hold beyond Llama and Qwen models?
- Why might API access to models produce better accuracy than consumer products?
- How should benchmark design account for task-dependent sparsity tolerance differences?
- What compute costs separate a panel of judges from a single large judge?
- Can parameter compression mechanically force value systems toward idealized centers?
- Why do metric choices constrain which model capabilities get developed?
- What output distribution properties make smaller models better for wide sampling?
- Are newer larger language models actually worse at faithful summarization?
- Which architectural choices matter most when a model must fit one billion parameters?
- Can compact extractors outperform large models in RAG pipelines?
- Can scaling predictions become reliable if improvements are continuous not sudden?
- How do input length constraints reshape personalization system design choices?
- How do conditional scaling laws incorporate hardware into architecture choices?
- Can depth scaling and breadth scaling unlock independent capability axes?
- How does sparsity tolerance vary across different task types?
- What mobile hardware constraints force the sub-billion parameter regime?
- How does the Ladder of Scales approach reduce search costs across model sizes?
- Do rare cultural concepts fail predictably as model scale increases?
- What power-law scaling patterns emerge when consistency models are trained at scale?
- What constraints force mobile deployments to operate in the sub-billion parameter regime?
- Why do power-law distributions make standard ML infrastructure assumptions fail?
- How should GPU execution paths and training objectives co-design sparsity?
- What are the scaling law differences between vision and language learning?
- Why are consumer GPU open models advancing faster on LM Arena metrics?
- How do orthogonal adapter vectors avoid interference at scale?
- Why does input embedding magnitude affect perturbation sensitivity in transformers?
- Why do production systems optimize for three model classes instead of foundation models?
- Why does depth outperform width for sub-billion parameter models?
- Can QLoRA match full precision results at 33B and 65B model scales?
- Do scaling laws change when weight precision becomes a design variable?
- Does the Chinchilla balance apply equally across all data types or only language?
- Do open-weight models dominate particular task categories over proprietary alternatives?
- What architectural variables make entropy-based patching work at 8B scale?
- How do power-law distributions differ from uniform collision assumptions?