If you ask several different AIs to judge something, do their built-in biases cancel out, or only the ones they don't share?
Does disjoint family diversity actually cancel model-specific bias in evaluation?
This explores whether building a judging panel from models made by different developers ('disjoint families') really washes out each model's built-in slant, or only the part of the slant the models don't share.
This explores whether a judging panel drawn from different model families really washes out each model's built-in slant, or only the part of the slant the models don't share. The corpus points to the second: it cancels the quirks that differ between models, not the bias they all inherit.
The case for cancellation is real. A panel of smaller judges from disjoint families beat a single large judge, showed less intra-model bias, and cost over 7× less. No single judge was best everywhere, which is what you'd expect if each has its own quirks that a panel averages out Can a panel of smaller judges outperform one large judge?. The same logic shows up in training. Models trained on many imperfect experts behave like a majority vote, which denoises errors that are uncorrelated across sources Can models trained on many imperfect experts outperform everyone?. Self-favoritism is a concrete model-specific bias. Claude shows a small pro-Anthropic tilt across four evaluation tasks, GPT only when grading agentic work, and Gemini a weak anti-Google tilt Do frontier AI models favor their own company?. Because the tilts differ in direction and size, mixing families dilutes any one company's thumb on the scale.
But everything hinges on the errors being uncorrelated. Cognitive biases in LLMs are planted mostly in pretraining. Models that share a pretrained backbone show similar bias patterns however they were finetuned Where do cognitive biases in language models come from?. So five finetunes of one base model make a panel of one. That is why 'disjoint families' is the right instinct. Yet different logos don't guarantee different diets. A study of 70+ models across 26K open-ended queries found an 'Artificial Hivemind': models independently produced strikingly similar or identical answers, traced to overlapping training data and alignment procedures Do different AI models actually produce diverse outputs?. Whatever bias lives in the shared text and shared preference-tuning recipes passes straight through the vote. Agreement among the judges then looks like confirmation.
Two more limits apply. Diversity isn't free. In multi-agent ideation, diverse teams without real expertise did worse than one competent agent Does cognitive diversity alone improve multi-agent ideation quality?, so each panelist has to be good enough to judge. And a third-party judge panel is one of the external anchors that self-improvement research says you need Can models reliably improve themselves without external feedback?. It anchors only as far as the panelists are independent of the evaluated model and of each other.
Disjoint families cancel the bias that varies between models, such as self-preference and house style. They don't cancel bias the models hold in common. The corpus shows reduced bias, not eliminated bias. It has no direct test of how much shared bias survives inside a judge panel, so 'cancels' is more than the evidence supports.
Sources 7 notes
PoLL (Panel of LLM evaluators) using multiple smaller models from disjoint families outperforms single large judges, reduces intra-model bias, and costs over 7× less. Across three settings and six datasets, no single judge was best everywhere, but diverse panels performed consistently well.
Models trained on diverse experts converge on consensus behavior that outperforms individuals. Low-temperature sampling concentrates outputs on this majority-voted consensus, denoising uncorrelated biases and errors across the training set.
Claude models show consistent small pro-Anthropic bias across four evaluation tasks, while GPT models show bias only in agentic grading, and Gemini shows weak anti-Google bias. The differences warn against treating company favoritism as universal.
A causal experiment using random-seed variation and cross-tuning showed that models sharing a pretrained backbone exhibit similar bias patterns regardless of finetuning data. Biases are planted during pretraining and merely swayed by instruction tuning.
INFINITY-CHAT analyzed 70+ models across 26K open-ended queries and found an "Artificial Hivemind" effect: models independently generate strikingly similar or identical responses due to overlapping training data and alignment procedures, undermining the diversity benefits of model ensembles.
Show all 7 sources
Multi-agent teams substantially outperform solo ideation, but only when members possess genuine senior knowledge. Diverse teams without expertise underperform even a single competent agent, because cognitive stimulation without expertise triggers process losses instead of insight.
Pure self-improvement stalls due to the generation-verification gap, diversity collapse, and reward hacking. Reliable improvement methods succeed by smuggling in external anchors: past model versions, third-party judges, user corrections, or tool feedback.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Planted in Pretraining, Swayed by Finetuning: A Case Study on the Origins of Cognitive Biases in LLMs
- Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)
- ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs
- NoveltyBench: Evaluating Language Models for Humanlike Diversity
- On Epistemic Diversity in Large Language Models
- Human diversity fuels collective creativity that large language models cannot simulate or sustain
- Transcendence: Generative Models Can Outperform The Experts That Train Them
- Beyond Brainstorming: What Drives High-Quality Scientific Ideas? Lessons from Multi-Agent Collaboration