A panel of small, diverse AI judges consistently outperforms one large expert model — and costs over seven times less.

This explores whether a panel of several smaller LLM judges cancels out evaluation bias better than one big judge — and the corpus has a fairly direct answer: yes, and cheaply. The clearest evidence comes from work on panels of smaller judges (PoLL), where an ensemble of small models drawn from *different model families* consistently beats a single large judge like GPT-4, introduces less intra-model bias, and costs over 7× less Can smaller models in panels outperform a single large judge?. The key isn't that small models are individually better — they aren't — it's that a single large judge carries its own family-specific quirks (favoring verbose answers, its own phrasing, its own house style), and there's no one to outvote it. Diversity across families cancels those correlated biases.

Why does averaging across diverse-but-mediocre evaluators beat one strong one? The corpus has a deeper principle hiding here. Models trained on many imperfect experts converge, through a kind of implicit majority vote, toward consensus that outperforms any individual expert — because uncorrelated errors denoise each other while the shared signal survives Can models trained on many imperfect experts outperform each one?. A judge panel is the same trick applied at evaluation time: each judge's idiosyncratic bias is noise, and noise that isn't shared across the panel washes out. That's also why panel *diversity* matters more than panel *size* — three judges from the same family mostly repeat each other's mistakes.

But ensembling isn't the only lever on judge bias, and this is where lateral reading pays off. A completely different line of work attacks bias inside a *single* judge by teaching it to reason during evaluation — converting judgment into a verifiable problem and training with RL so the judge thinks through its decision instead of latching onto surface features like authority, verbosity, or position Can reasoning during evaluation reduce judgment bias in LLM judges?. Related work shows generative judges that produce reasoning chains about each step outperform classifier-style reward models, and do it with far less training data Can judges that reason about reasoning outperform classifier rewards?. So the field has two roads to less-biased judgment: make the judges *diverse* (ensemble) or make each judge *deliberate* (reasoning). They're complementary — nothing stops you from ensembling reasoning judges.

There are also cheaper, mechanical fixes worth knowing about. One is that many judge "biases" surface as ties and coarse discrete scores; reading the expectation over the scoring-token logits turns judgment into a continuous score that sharpens discrimination without any extra models Can reading logit distributions break ties in LLM judging?. Another framing worth carrying over: diverse mediocre contributors can beat redundant strong ones when an aggregator has raw material to arbitrate — the argument behind rewarding a *set* of traces for collective usefulness rather than each trace's individual correctness Can diverse mediocre traces outperform redundant expert traces?. That's the same intuition that makes a judge panel work, just pointed at generation instead of evaluation.

The thing you might not have expected: the win from ensembling isn't really about being "smarter" — it's a bias-cancellation effect that only shows up when the judges are *different from each other*. That reframes the practical question. Don't ask "how big should my judge be?" Ask "how uncorrelated are my judges' mistakes?" — and if you can't get diversity, make your single judge reason its way past the surface features it would otherwise be fooled by.