Line of inquiry
Inquiring lines›How do training data and architect…›What does iterative improvement ac…›this line of inquiry
How does model scale affect fine-tuning and downstream task performance?
A broader line of inquiry — a family of 119 specific questions the research asks around this. Follow one into its inquiring-line page, or move sideways to a related line below.
Questions in this line of inquiry 119
Specific inquiring lines the field asks around this — ordered from the most general framing down to the most specific angle.
- Does fine-tuning a small model match fine-tuning a large one?
- Can smaller models actually perform well on specific downstream tasks?
- Does fine-tuning actually change model capabilities or only output distribution?
- Can externalized memory and skills replace model scaling?
- How does model scale affect the crossover point between base and post-trained performance?
- Do small models show different parameter efficiency patterns than large models?
- Can smaller models produce skill updates as useful as frontier model updates?
- Why do smaller and larger models converge on different output formats?
- Do smaller models match Chinchilla-optimal performance when trained much longer?
- How should tiny language models be architected differently than large ones?
- Can frontier-scale results be predicted from small-scale benchmark speedups?
- Why do larger models reduce interference between rare and common tasks?
- Can RL format selection explain performance gains attributed to algorithmic improvements?
- Does small heterogeneous model architecture outperform large homogeneous pools economically?
- Does LoRA's performance scale depend on model architecture or domain?
- Does parameter isolation per task enable online updates without retraining?
- Does pretraining data size matter less than base model scale for finetuning?
- Can smaller specialist models outperform large generalist models on domain tasks?
- How do retrieval and fine-tuning trade off flexibility against training cost?
- How do finetuning and pretraining improvements differ in their effects on model capabilities?
- Why do model capability gains appear to regress between major releases?
- Can smaller language models match large model accuracy at lower total cost?
- What capabilities actually require massive scale versus specialized training regimes?
- What performance trade-offs emerge when composing multiple independently trained model capabilities?
- Does the optimal model size depend on what capabilities you actually need?
- What role does inductive bias play versus model capacity in practice?
- Does memory granularity need to match the task domain or the model size?
- Do all models of equivalent quality see the same inference request volume?
- Why do scaling laws show capability saturation at specific thresholds?
- How much do cross-model improvements compare to the original performance gains on the training benchmark?
- What hidden costs emerge when you fine-tune models for a single domain?
- Why do fine-tuned models fail outside their specialized domains?
- How can smaller models help select useful data for larger models?
- Can task type alone predict which model size is sufficient?
- What happens to the optimal model size as projected inference volume grows?
- How do larger models maintain more parallel tasks than smaller models?
- Which model capabilities actually matter for sustained workflow delegation?
- Why do parameter-based compressors fail to measure true model simplicity?
- What capability risks emerge when models are optimized for single domains?
- Why do scaling laws fail to predict optimal architectures at small parameter counts?
- How small must the anchoring stream be to correct world model bias?
- Can expert vectors learned offline transfer across multiple model architectures?
- Why do metric choices constrain which model capabilities get developed?
- How does the hardware-aware scaling recipe enable looped models to win?
- Why do small language models show faster on-premise payback than large ones?
- What makes frozen model reasoning different from weight-based parameter updates?
- Which finetuning method works best across different task and data regimes?
- Is model selection a stronger security lever than improving individual model defenses?
- Are cheap testbeds and skewed task distributions linked by design necessity?
- What makes a small surgical wide component sufficient with a capable deep model?
- Does the O(1)-bit signal theory hold beyond Llama and Qwen models?
- How can expensive models efficiently support cheap models in production?
- What production constraints should determine paradigm selection?
- Why do only two of fourteen models improve when problem constraints are removed?
- Why does adjusted compression performance degrade as models scale larger?
- How does model size within a family affect output distinguishability?
- Why does parameter-efficient tuning scaling fail to improve finetuning performance?
- What causes models to develop domain capability cliffs after specialization?
- Can a complexity-predictor be meaningful if models are redundant?
- Do different domains require different types of model investment?
- Can parameter compression mechanically force value systems toward idealized centers?
- What happens to base model capabilities when you apply finetuning?
- Are larger models and search access substitutes for factual accuracy?
- Why do some model variants deceive while their sibling models refuse?
- Does preference tuning help or hurt the exploration of solution spaces in code?
- Are architectural differences between models more detectable than size differences?
- Can ensemble predictions be distilled back into a single deployable model?
- How much performance is lost when converting pretrained checkpoints versus training from scratch?
- How does over-specialization create capability cliffs outside target domains?
- What task structures benefit most from geometric parameter merging?
- What compute costs separate a panel of judges from a single large judge?
- How do ensemble methods apply within a single model?
- What is minimum viable scale for deploying an AI system?
- Can specialized components replace single fully-trained models in deployment?
- Can depth scaling and breadth scaling unlock independent capability axes?
- What filtering criteria best identify student-compatible refinements from teacher models?
- Can scaling predictions become reliable if improvements are continuous not sudden?
- Which architectural choices matter most when a model must fit one billion parameters?
- Can per-user adapters remain consistent without drifting or leaking?
- What output distribution properties make smaller models better for wide sampling?
- How does the Ladder of Scales approach reduce search costs across model sizes?
- At what dataset size does LoRA performance start degrading?
- Why do production systems optimize for three model classes instead of foundation models?
- What mobile hardware constraints force the sub-billion parameter regime?
- How do optimal learning rates differ between LoRA and full fine-tuning?
- How do virtual model instances preserve identity through load-balancing and failover?
- Why are consumer GPU open models advancing faster on LM Arena metrics?
- How quickly do open models close the performance gap after closed model releases?
- What access constraints allow description-based adaptation but block conventional techniques?
- Why do power-law distributions make standard ML infrastructure assumptions fail?
- Can QLoRA match full precision results at 33B and 65B model scales?
- What constraints force mobile deployments to operate in the sub-billion parameter regime?
- Do compression guidelines optimized for GPT models transfer to other LLM families?
- How does model capability relate to personality conditioning flexibility?
- When should full-parameter post-training be used instead of LoRA adaptation?
- How do RL subnetworks identified from different random seeds compare?
- Which open source models meet the threshold for viable on-premise deployment?
- Do open model properties like customizability create net new misuse opportunities?
- Does the Chinchilla balance apply equally across all data types or only language?
- Why does depth outperform width for sub-billion parameter models?
- Do open-weight models dominate particular task categories over proprietary alternatives?
- How does QLoRA differ from training ternary weights natively from scratch?
- Why does placing LoRA adapters at every layer matter for QLoRA performance?
- How do spectral-norm constraints prevent divergence in world model rollouts?
- How do trait adapters interact with different base model architectures?
- Why does verification sit on a different scaling axis than pre-training?
- Do scaling laws change when weight precision becomes a design variable?
- How do pretraining token counts affect downstream performance beyond typical ratios?
- Can different teams train specialist adapters that share one frozen base model?
- Why does OpenAI believe alignment monitoring requires scaling alongside model capability?
- What architectural variables make entropy-based patching work at 8B scale?
- How much does pretraining quality affect the modularity of fine-tuned models?
- What causes Chinchilla scaling curves to flatten with extreme training data?
- How do aligned LoRA adapters compose through parameter-space arithmetic?
- How does Easy Consistency Tuning accelerate consistency model training from diffusion checkpoints?
- What are the five inseparable design choices when building world models?
- How does remote object store throughput affect KV cache loading latency?
- What role do CM fields play in constructing high unit distance configurations?
- What makes two timescales better than one for minimizing weight movement?