Why aren't bigger models better for generating diverse outputs?
When generating many unique outputs within a fixed budget, does model size actually matter? Exploring whether the conventional wisdom of using larger models holds for diversity-focused tasks.
A non-obvious finding from Evaluating the Diversity and Quality of LLM Generated Content: when the goal is to generate as many unique outputs as possible within a fixed sampling budget — the canonical use case for synthetic data generation — smaller models around 500M parameters are often the most efficient choice. Larger models do not produce proportionally more unique outputs per sample.
This is consistent with a broader pattern. Larger models concentrate probability mass more tightly on their preferred outputs. They are less likely to sample widely across the output space and more likely to repeat their top candidates. For applications that want diversity within the cost of a fixed inference budget, this concentration is a liability. A smaller model with flatter output distributions can produce a broader spread of distinct outputs at the same compute cost.
The "use the biggest model you can afford" heuristic comes from tasks where each output is consumed individually — answering one question, writing one summary. For those tasks, output quality dominates and bigger is better. The synthetic-data and unique-content regime inverts the calculus. Each output contributes proportional to its distinctiveness from the others; aggregate value is set by the variance of the sampled set, not by the peak quality of any single output.
For practitioners building synthetic training data, this argues for choosing model size based on the value function. If quality per output matters most, use the largest model. If uniqueness per dollar matters most, drop to a smaller model and run more samples. The crossover point varies by task; the paper places it around 500M parameters for programming tasks but the principle is general.
The deeper observation is that "model capability" is not a single scalar that monotonically rewards scale. Different applications care about different summary statistics of the output distribution, and those summary statistics scale differently. Treating scale as universally beneficial is an artifact of evaluating models on single-output benchmarks.
Inquiring lines that read this note 38
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
When does architectural design matter more than raw model capacity?- When does the right constraint beat additional model capacity?
- What production constraints should determine paradigm selection?
- How do larger models maintain more parallel tasks than smaller models?
- Does the optimal model size depend on what capabilities you actually need?
- Why does depth outperform width for sub-billion parameter models?
- How does the Ladder of Scales approach reduce search costs across model sizes?
- Why do production systems optimize for three model classes instead of foundation models?
- Do small models show different parameter efficiency patterns than large models?
- Which architectural choices matter most when a model must fit one billion parameters?
- What makes a small surgical wide component sufficient with a capable deep model?
- How can expensive models efficiently support cheap models in production?
- Can few-shot examples narrow generative diversity in creative tasks?
- How do quality, diversity, and complexity create different effects on downstream model performance?
- How does graph-based tool sampling differ from random sampling in diversity?
- How do quality thresholds change which model produces more usable diversity?
- How should we evaluate diversity differently across programming and creative tasks?
- What makes creative writing diversity different from code diversity fundamentally?
- How does probability mass concentration affect sampling diversity across model scales?
- At what point does output quality outweigh diversity value in synthetic data tasks?
- How do complexity and diversity affect model performance differently?
- How do you identify which models should form a minimal diverse coreset?
- Do different domains require different types of model investment?
- Why do smaller and larger models converge on different output formats?
- Does fine-tuning a small model match fine-tuning a large one?
- How can smaller models help select useful data for larger models?
- What output distribution properties make smaller models better for wide sampling?
- Can multiple small models outperform a single large model with good routing?
- Why might diverse smaller models with routing beat one giant model?
- Does small heterogeneous model architecture outperform large homogeneous pools economically?
Related concepts in this collection 2
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does preference tuning actually reduce the diversity of model outputs?
The field assumes RLHF and DPO reduce diversity, but this assumption rests on measuring all outputs equally. What happens if we only count diverse outputs that meet quality thresholds?
same paper, the broader diversity-metric reframing
-
Does preference tuning always reduce diversity the same way?
Explores whether the standard narrative that RLHF reduces model diversity holds equally across different task domains, or if the effect varies by what the domain rewards.
same paper, the domain-dependence
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Scaling Synthetic Data Creation with 1,000,000,000 Personas
- Personalized Dialogue Generation with Persona-Adaptive Attention
- Reasoning-Driven Synthetic Data Generation and Evaluation
- Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
- Foundation Priors
- Orchestrating Synthetic Data with Reasoning
- Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models
- Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains
Original note title
smaller models around 500M parameters are most efficient for unique-output generation within a fixed sampling budget — parameter scale is not monotonic