Training an AI on many sources can cancel out each one's mistakes, but only if they don't all err alike.
Why does diversity in training data enable denoising rather than reinforce shared biases?
This explores why training on many different sources can cancel out individual mistakes instead of averaging in everyone's prejudices, and when that cancelling stops working.
This explores why training on many different sources can cancel out individual mistakes instead of averaging in everyone's prejudices, and when that cancelling stops working. The corpus suggests the answer is conditional: diversity denoises only when the sources make *different* errors. It does nothing for errors they all share.
The clearest case is a model trained on many imperfect experts. Each expert's slips and quirks are mostly their own, so the model learns something close to the consensus. Low-temperature sampling, which leans toward the most likely answer, lets that consensus win out. That works like an implicit majority vote, and the result can outperform every individual expert Can models trained on many imperfect experts outperform everyone?. The important word is *uncorrelated*. One expert's mistake gets outvoted because the others don't make it. Diversity matters because it points the errors in different directions so they cancel.
Shared bias is the opposite case. If every source makes the same mistake, the vote is unanimous and majority voting locks the mistake in. The 'Artificial Hivemind' study shows how easily this happens: across 70+ models and 26K open-ended queries, different LLMs gave strikingly similar or even identical answers, because their training data and alignment procedures overlap Do different AI models actually produce diverse outputs?. A group of models looks like many voters but behaves more like one voter cloned, so ensembling them denoises much less than you'd hope. A related warning is that high accuracy can hide a systematic error underneath Can AI models be truly free from human bias?. A consensus that scores well is not proof that it's right.
The ingredient can also be lost after pretraining. In controlled experiments, RL post-training amplified a single pretraining format within the first epoch and collapsed the alternatives Does RL training collapse format diversity in pretrained models?. Preference tuning is less uniform. It reduces lexical and syntactic diversity in code but increases it in creative writing, because each domain rewards something different Does preference tuning always reduce diversity the same way?. Whether the denoising ingredient survives therefore depends on what the training rewards.
Some work treats protecting diversity as a deliberate lever. Step-level critique during training counteracts the narrowing of the solution tail and keeps solutions varied across self-training rounds Do critique models improve diversity during training itself?. Explicitly rewarding semantic diversity improved both variety and quality on creative and math tasks Can diversity optimization improve quality during language model training?. The better question is less 'is my data diverse?' and more 'are my sources' errors independent?' Nothing in this set of notes shows how to measure that independence directly, and that gap is where the hivemind result becomes worrying.
Sources 7 notes
Models trained on diverse experts converge on consensus behavior that outperforms individuals. Low-temperature sampling concentrates outputs on this majority-voted consensus, denoising uncorrelated biases and errors across the training set.
INFINITY-CHAT analyzed 70+ models across 26K open-ended queries and found an "Artificial Hivemind" effect: models independently generate strikingly similar or identical responses due to overlapping training data and alignment procedures, undermining the diversity benefits of model ensembles.
Research shows that 'theory-free' AI models mask bigotry behind high accuracy metrics while committing fundamental statistical errors. A 95% accurate criminal justice system would wrongly convict thousands, demonstrating that model sophistication does not validate causal inference.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
RLHF reduces lexical-syntactic diversity in code generation but increases it in creative writing. The direction depends on what each domain incentivizes: code rewards convergence toward correct solutions, while creative writing rewards stylistic distinctiveness.
Show all 7 sources
Step-level critique in the training loop counteracts tail narrowing and maintains solution diversity across self-training iterations. This training-time benefit—preventing premature convergence—is more fundamental than test-time accuracy gains.
DARLING jointly optimizes for quality and semantic diversity using a learned classifier, finding that diversity rewards catalyze exploration and produce higher-quality outputs than quality-only baselines across both creative and mathematical tasks.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Evaluating the Diversity and Quality of LLM Generated Content
- Jointly Reinforcing Diversity and Quality in Language Model Generations
- NoveltyBench: Evaluating Language Models for Humanlike Diversity
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)
- Outcome-based Exploration for LLM Reasoning
- Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs
- Vector Policy Optimization: Training for Diversity Improves Test-Time Search