Could a team of smaller AIs grading answers together cost far less than paying one giant AI to do it?
How much inference compute does panel-of-judges evaluation actually cost?
This explores what it costs, in inference compute, to have several LLMs jointly judge an output instead of one big model, and what makes that bill go up or down.
This explores what a panel of LLM judges costs in inference compute compared with a single large judge, and what pushes that bill up or down. The corpus has one hard number: a panel of smaller judges drawn from different model families cost over 7× less than a single large judge, and it also showed less bias toward its own model family. Across three settings and six datasets, no single judge was best everywhere, but the diverse panels were consistently good Can a panel of smaller judges outperform one large judge?. The likely reason is that several small models can cost less in total than one big one. That reasoning is mine, not the note's, and the note doesn't break the cost down by panel size or token count.
That 7× figure assumes each panel member gives a quick verdict. The cost changes if judges reason before scoring. Several teams found that adding chain-of-thought before a reward score lets evaluation spend more compute where it's needed Can reward models benefit from reasoning before scoring?. Judges trained to reason also resist bias from authority, verbosity, position and beauty Can reasoning during evaluation reduce judgment bias in LLM judges?. Judges that write out step-by-step reasoning beat classifier-style ones with far less training data Can judges that reason about reasoning outperform classifier rewards?. You pay for more tokens per verdict, and you multiply that by the number of panelists. The corpus doesn't price a panel of reasoning judges.
Three notes suggest how to keep that cost down. First, spending the same total compute unevenly, with less on easy prompts and more on hard ones, beats spending it uniformly Can we allocate inference compute based on prompt difficulty?. For a panel, that could mean asking one cheap judge first and calling the rest only when it's unsure. That application is my extrapolation, not something the note tests. Second, smaller models given more inference compute can match larger ones on hard prompts, so pretraining and inference compute are partly interchangeable Can inference compute replace scaling up model size?. A panel is one way to spend that trade. Third, in one verification pipeline the LLM judge used most of the budget, while mechanical checks like parse validation cost orders of magnitude less How much do deterministic guardrails actually cost to run?. Running cheap deterministic checks before any judge sees the output avoids paying panel prices for outputs that were already broken.
The heavier the evaluator, the less cost information the corpus gives. An agentic judge that collects evidence cut judge shift to 0.27% from 31% for a plain LLM judge, but its memory module cascaded errors Can agents evaluate AI outputs more reliably than language models?. An inference-scaled reviewer that checks proofs and experiments line by line caught flaws that passed human review at STOC and ICML Can inference scaling help reviewers catch errors humans miss?. Neither summary reports a compute bill, so I can't say what those gains cost.
More judges also won't help if they aren't the right kind. Reasoning-trained models keep beating non-reasoning ones however many tokens the non-reasoning model is given Can non-reasoning models catch up with more compute?. So the corpus supports two claims: a diverse panel of small judges can cost several times less than one large judge, and reasoning, agentic or adaptive judging makes each verdict costlier and better. It doesn't yet say what a panel that combines both would cost.
Sources 10 notes
PoLL (Panel of LLM evaluators) using multiple smaller models from disjoint families outperforms single large judges, reduces intra-model bias, and costs over 7× less. Across three settings and six datasets, no single judge was best everywhere, but diverse panels performed consistently well.
Three independent teams (RRM, RM-R1, DeepSeek-GRM) discovered that adding chain-of-thought reasoning before reward scoring enables adaptive test-time compute scaling for evaluation. Reasoning-based approaches raise the capability ceiling of reward models beyond what outcome-based evaluation achieves.
Training judges with reinforcement learning to reason about evaluations—by converting judgment tasks into verifiable problems with synthetic data pairs—produces judges that think through their decisions rather than relying on exploitable surface features, directly mitigating authority, verbosity, position, and beauty bias.
StepWiser demonstrates that training judges to produce reasoning chains about policy reasoning—rather than classify steps—yields better judgment accuracy and data efficiency. Independent confirmation from GenPRM and ThinkPRM shows generative PRMs outperform discriminative ones with orders of magnitude less training data.
Research shows inference effectiveness varies dramatically by prompt difficulty. Reallocating the same total compute adaptively—giving easy prompts less and hard ones more—substantially outperforms larger models under uniform budgets.
Show all 10 sources
Snell et al. (2024) showed that inference-time compute trades off against model parameter scaling, especially on difficult prompts. This reveals pretraining and inference compute are not independent resources.
Mechanical checks like parse validation and planted cases require only arithmetic and string operations, while judges consume most budget on inference. This cost gap makes verification layers realistic without competing for computational resources.
Eight-module agentic evaluation achieved 0.27% judge shift versus 31% for LLM-as-a-Judge on complex tasks. However, the memory module cascaded errors, revealing that agentic systems need error isolation mechanisms to maintain gains.
PAT, an agentic reviewer using test-time compute to check proofs and experiments line by line, achieves 34% better recall on math errors than zero-shot approaches and surfaced critical flaws at STOC and ICML that passed human review.
Reasoning models persistently outperform non-reasoning models regardless of inference budget because training instills a reasoning protocol that makes additional tokens productive. The gap is fundamentally about deployment mechanisms and training structure, not raw capability.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge
- J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning
- Reasoning Models Can Be Effective Without Thinking
- When More Thinking Hurts: Overthinking in LLM Test-Time Compute Scaling
- Think Twice: Enhancing LLM Reasoning by Scaling Multi-round Test-time Thinking
- ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
- RM-R1: Reward Modeling as Reasoning
- Reward Reasoning Model