Hiring a committee of cheaper AI judges beats relying on one expensive expert — and they're less biased too.

This reads the question literally: does a *panel* of smaller LLM judges actually beat a single large judge, and by how much? The corpus has a direct answer. The PoLL work (Panel of LLM evaluators) found that a panel of smaller models drawn from different families consistently outperforms a single large judge like GPT-4 — while introducing less intra-model bias and costing over 7× less Can smaller models in panels outperform a single large judge?. So the honest headline isn't 'a few points better': it's roughly comparable-to-better accuracy at a fraction of the cost, with the real win being bias reduction rather than raw score.

The mechanism is worth pausing on, because it's the interesting part. The panel doesn't win by being collectively smarter — it wins because a single judge carries family-specific bias (it favors outputs that look like its own family's style), and pooling judges from *different* model families cancels those biases against each other Can smaller models in panels outperform a single large judge?. This is a diversity effect, not a scale effect. That framing connects to a surprising cousin finding: smaller models (around 500M params) actually generate *more* unique outputs per sample than larger ones, because bigger models concentrate probability mass on their preferred answers Why aren't bigger models better for generating diverse outputs?. Bigness buys confidence, and confidence narrows variety — which is exactly the raw material a panel needs.

That same logic shows up in reinforcement learning, where 'set' rewards optimize a batch of traces for collective usefulness rather than per-trace correctness — diverse mediocre traces beat redundant strong ones because an aggregator needs disagreement to arbitrate, not confirmation Can diverse mediocre traces outperform redundant expert traces?. A panel of judges is the evaluation-side version of the same bet: you want complementary errors, not one very confident error.

But the corpus also tells you where the panel advantage *shrinks*, which is what keeps this from being a free lunch. Multi-agent advantages narrow as single-agent capability improves — when the base model gets strong enough, a lone agent often matches or beats the committee, and the panel's coordination overhead stops paying for itself When do multi-agent systems actually outperform single agents?. And there's a second lever that competes with 'add more judges': you can instead make a single judge *think*. Training a judge with RL to reason through its evaluation — turning judgment into a verifiable problem — directly reduces authority, verbosity, position, and beauty bias, the very biases a diverse panel was cancelling Can reasoning during evaluation reduce judgment bias in LLM judges?.

So the sharpest answer isn't a number — it's a trade-off map. A panel of smaller judges is cheaper (7×+), less biased, and roughly as accurate as one large judge *today*, and it wins specifically because bias cancels across model families. But that edge erodes as individual judges get stronger, and there's a rival strategy — a single reasoning-trained judge — that attacks the same bias problem from the inside. If you want an even cheaper lever, reading the expectation over a judge's scoring-token logits produces continuous scores that break ties without adding any models at all Can reading logit distributions break ties in LLM judging?. The panel is one good tool; the reason it works tells you when to reach for a different one.