LLM-as-a-Judge Is Not an Oracle: Why Self-Improving Agents Need Deterministic Guardrails

Observed failure modes of LLM evaluators under optimization pressure, and the guardrails that contained them.

Introduction. Consider a concrete setup, one of the two we describe in this paper. An LLM is given a codebase and asked to rate it from 1 to 5 on qualities such as readability and robustness. Expert human reviewers have already rated the same codebases, so we can measure how far the model's ratings sit from theirs: a mean absolute error of 0.96 means the model is typically off by roughly one point on the five-point scale. To close that gap without a human in the loop, a second LLM acts as an optimizer. It repeatedly rewrites the judge's instructions, re-runs the judge, and keeps whichever version of the instructions scores best. In an early prototype of ours, the optimizer improved the judge by deleting it. The mutation it proposed replaced the entire scoring rubric with a placeholder string. The judge, now with nothing to grade against, returned unstructured prose containing none of the expected rating fields. Our evaluation harness caught the resulting parsing errors and quietly fell back to a default rating of 3 on every dimension.

Discussion / Conclusion. The failures in this paper are not arguments against using LLMs to evaluate. They are arguments about where an LLM's verdict belongs in a system that acts on it. An evaluator that is occasionally wrong is a perfectly serviceable component; an evaluator that is occasionally wrong and holds final authority over an optimizer running thousands of iterations is a liability, because the optimizer will find the cases where it is wrong and build on them. The difference is not the model's accuracy. It is the position the model occupies. That position is the one thing in the design that is genuinely ours to choose. We cannot make a judge unbiased, and Section 4.3 suggests we cannot even reliably make it less biased by asking. What we can do is arrange the system so that being wrong is survivable: put the checks that cannot be argued with in front of the checks that can, measure correctness rather than asserting it, keep a partition of the data where nothing that proposes changes can see it, and plant cases whose success is itself the alarm. None of these are sophisticated. Their value lies precisely in being mechanical, and Section 5.7 makes the pleasant observation that they are also, by orders of magnitude, the cheapest part of the loop to run.