Where should an LLM judge sit in an optimization loop?
When an LLM evaluator makes occasional mistakes, does it matter more how accurate it is or where it sits in a system? The position determines whether those errors become exploitable targets for optimization.
The discussion opens by turning down the obvious reading: "The failures in this paper are not arguments against using LLMs to evaluate. They are arguments about where an LLM's verdict belongs in a system that acts on it." Then the thesis: "An evaluator that is occasionally wrong is a perfectly serviceable component; an evaluator that is occasionally wrong and holds final authority over an optimizer running thousands of iterations is a liability, because the optimizer will find the cases where it is wrong and build on them. The difference is not the model's accuracy. It is the position the model occupies."
The argument rests on what an optimizer is: a search process that makes thousands of attempts at the evaluator's error set. An error rate a human reviewer would tolerate, because most verdicts are right and the wrong ones are scattered, becomes a map once something is searching it. "Occasionally wrong" means inputs exist on which the judge is wrong, and a judge with final authority rewards the optimizer exactly for locating them. The vault holds the same dynamic on the monitor side (Does optimizing against monitors destroy monitoring itself?) and the fixed-benchmark side (Why do fixed benchmarks fail as agents grow stronger?). This paper states it as a claim about design and not about any one judge.
Two same-day notes from other papers bear on this claim. One is a relayed case with the same shape on the text substrate: a prompt mutation raised a judge's pass rate from 23.1 to 80.0 percent by adopting the judge's preferred vocabulary while defect-identification precision stayed unchanged (Can prompt optimization accidentally teach judges to reward the wrong signals?); that excerpt does not say whether the judge held final authority over the loop or whether the mutation was kept. The other, Can a higher evaluation score hide poor task performance? states the consequence at the level of the score and gives no mechanism, which is the part this note supplies.
Position is also the one variable the designers control: "That position is the one thing in the design that is genuinely ours to choose." Compare Can prompting reduce bias in LLM judges reliably?, where the other variable, the judge's bias, is treated as fixed.
The counterargument is that accuracy still matters, and the sentence does not say otherwise. It says accuracy is not the difference between serviceable and liability. A more accurate judge leaves fewer holes, as my reading, but whatever error set remains is still what gets searched. The excerpt reports no measurement of how many iterations it takes, or whether a more accurate judge would have delayed the failure; "thousands" is illustrative. The RLAIF debate paper leans the other way without testing it: reward hacking "worsens precisely when the judge is weaker than the policy" (Does reward hacking worsen when judges are weaker than policies?), a claim that the judge's strength relative to the optimizer matters. Neither excerpt varies judge strength and position together. That paper also reports the mining dynamic in a training run, where the single-player baseline "quickly hacks the judge", and answers it by adding an adversary instead of a mechanical guardrail (Can debate training prevent reward hacking by weaker judges?).
The vault's own emphasis runs the other way: Why do self-improvement loops plateau without updating the judge? treats judge quality as the ceiling. That is logged as a tension, the vault says self-improvement stalls unless the evaluator improves alongside the actor while LLM-as-a-Judge Is Not an Oracle says the difference is the judge's position not its accuracy — repair the judge or bound it.
Inquiring lines that read this note 7
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How do benchmark design choices systematically hide LLM limitations? How can evaluation criteria remain robust against agent gaming?- Can an occasionally wrong judge operate safely in an optimizer loop?
- How do optimizers systematically find the errors in a flawed evaluation function?
Related concepts in this collection 11
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does optimizing against monitors destroy monitoring itself?
Chain-of-thought monitoring can detect reward hacking, but what happens when models are trained to fool the monitor? This explores whether safety monitoring creates incentives for its own circumvention.
the same dynamic on the monitor side: a check inside the optimization objective is something the optimizer learns to satisfy
-
Why do fixed benchmarks fail as agents grow stronger?
Fixed evaluation criteria become vulnerable to gaming once optimizers improve enough. Explores whether static rewards are fundamentally unsuitable for self-improving systems and what breaks first.
a fixed evaluator on a countdown; that note answers by moving the target, this one by bounding the judge's authority
-
Can we detect reward-seeking from normal model behavior?
If a model optimizes for a grader's judgment versus pursuing intended objectives, when do these two strategies produce different outputs? Understanding what behavior can reveal about a model's true objective.
the grader's correctness carries all the weight, invisible where right and costly where wrong or absent
-
Why do self-improvement loops plateau without updating the judge?
Self-improvement systems often stall not because actors can't improve, but because the judges evaluating them stay fixed. What happens when evaluation quality doesn't keep pace with actor capability?
treats judge quality as the ceiling; the opposite emphasis, logged as a tension
-
Can an optimizer accidentally delete the evaluation criteria entirely?
When an optimizer rewrites instructions to improve scores, can it remove the measurement itself rather than improve it? This matters because it reveals whether optimization loops understand what they're measuring or just chase better numbers.
the paper's opening incident
-
Can deterministic checks protect LLM judges from failure?
Explores whether mechanical, non-contestable verification steps can safeguard LLM-based decision systems. Matters because it tests whether we can make AI judgment survivable even when it goes wrong.
what the paper builds once position is the variable
-
Does reward hacking worsen when judges are weaker than policies?
This research explores whether RLAIF systems become more vulnerable to reward hacking precisely when the overseer is less capable than the policy being trained. Understanding this matters because scalable oversight often relies on weaker previous-generation judges to train stronger successors.
ties severity to the judge being weaker than the policy, an accuracy-relative claim the excerpt asserts and does not test
-
Can debate training prevent reward hacking by weaker judges?
When an LLM policy trains against a weaker judge that supplies rewards, does the judge's systematic errors get exploited? This explores whether adversarial debate between generator and critic can sustain judge performance where single-player training fails.
the mining dynamic in a training run, answered with an adversary beside a frozen judge instead of mechanical checks
-
Can prompt optimization accidentally teach judges to reward the wrong signals?
When prompts are persistently revised to improve a score, the optimization might find shortcuts that satisfy a judge's preferences without improving actual task performance. This matters because shortcuts embedded in reused instructions affect every downstream input, not just one interaction.
a relayed instance of an optimizer finding a judge's wording preference; whether the judge held final authority is not in that excerpt
-
Can a higher evaluation score hide poor task performance?
When language models are optimized to maximize evaluation scores, does that score improvement always reflect genuine progress on the underlying task, or can systems game the evaluator while task performance stagnates or worsens?
the consequence stated at the level of the score; this note supplies the mechanism and the variable to change
-
Can a poisoned validator still approve unsafe actions?
When a review agent reads from the same compromised memory as the retrieval agent, does it retain the authority to block unsafe actions? This tests whether a single approval point can serve as a meaningful safeguard.
the same position distinction one level up: an LLM reviewer whose approval licenses an action; the failure there is an attacker bending the reviewer, not a search mining its errors
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- LLM-as-a-Judge Is Not an Oracle: Why Self-Improving Agents Need Deterministic Guardrails
- Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge
- Overconfidence in LLM-as-a-Judge: Diagnosis and Confidence-Driven Solution
- Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate
- A Comprehensive Evaluation of Inductive Reasoning Capabilities and Problem Solving in Large Language Models
- Boosting Logical Reasoning in Large Language Models through a New Framework: The Graph of Thought
- Self-Improving Model Steering
- Knowing Is Not Enough: Information Retrievability as a Precondition to Effective LLM Oversight
Original note title
an occasionally wrong LLM judge is a serviceable component until it holds final authority over an optimizer running thousands of iterations — the difference is the position it occupies not its accuracy