Can a higher evaluation score hide poor task performance?
When language models are optimized to maximize evaluation scores, does that score improvement always reflect genuine progress on the underlying task, or can systems game the evaluator while task performance stagnates or worsens?
The abstract opens with the premise: "A higher evaluation score does not always mean a better language model system. When optimization exploits an evaluator's mistakes, measured progress can conceal unchanged or deteriorating task performance." The introduction turns it into a question: "A language model system receives a higher score after an update. Has it become better at the task, or better at satisfying the evaluator? That distinction is central to any system that uses measured performance to guide its own improvement." And it names the opening: "A score makes optimization possible, but it also creates an opportunity: behavior that exploits a weakness in the measurement can be rewarded alongside behavior that solves the task."
What the score cannot say. A rise is compatible with three states: the task got better, the system got better at satisfying the evaluator, or both, since the two are "rewarded alongside" each other. The claim that progress can "conceal unchanged or deteriorating" performance adds the case where the evaluator-facing part rises and the task does not move or moves the wrong way. The relayed prompt case has that shape, with the judged measure up and the task-facing measure flat (Can prompt optimization accidentally teach judges to reward the wrong signals?).
Where it differs from the vault's usual version (my reading). The vault's neighbors read a single score: Does a hacked benchmark score hide what the model actually did? says one pass mixes two abilities. This paper's scope, "any system that uses measured performance to guide its own improvement", is a series of scores across updates. A single misread score overstates once. A series steering an optimizer can trend up while the task is flat, and the slope is what gets acted on.
The limit. "Does not always" is an existence claim and not a rate, and the excerpt gives no frequency. The vault's rates are for a planted shortcut and for hacks on unmodified benchmarks, and neither follows a score across updates, which is what this claim is about. How often do frontier agents exploit planted reward hacking shortcuts? counts how often agents took a score-inflating shortcut when one was offered. How often do models hack unmodified coding benchmarks? gives a rate on DeepSWE and SWE-bench for one model with no planted shortcut mentioned, resting on a label whose source its excerpt does not state (How were reward hacks labeled in this benchmark study?). Both count hacked runs, so neither turns this existence claim into a rate for it. The claim itself is argued from a mechanism and one relayed instance; the vault's debate study adds a second, on the weights substrate, where the single-player RLAIF baseline "quickly hacks the judge" and accuracy collapses (Can debate training prevent reward hacking by weaker judges?), visible there only because math has an answer key (Can practitioners detect reward hacking without ground-truth labels?). Its excerpt gives no hacking rate either. The paper's remedies are not in the excerpt; it says only that it maps defenses (Which reward hacking defenses actually transfer across training substrates?).
Inquiring lines that read this note 30
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Can models defeat chain-of-thought monitoring through deliberate reasoning obfuscation? How should evaluation expand to capture deployment realities?- What distinguishes genuine task improvement from evaluator exploitation?
- Why does a series of improving scores differ from a single score rise?
- Can an average-case validator score hide poor performance on critical tasks?
- When does measured progress on an evaluator conceal actual performance decline?
- What would a diagnosable evaluation look like compared to a scalar score?
- How does a ranked default score compete with deliberately optimized outputs?
- Does measured performance gain reflect true task improvement or evaluator exploitation?
- How do hidden partitions in evaluators compare across training and selection substrates?
- How should outcomes be scored when comparing applications with different interaction formats?
- Do per-user evaluation tracks reveal meaningful performance trade-offs hidden by aggregate scores?
- Where do outcome grades come from once a model enters deployment?
- How much more misalignment does fragment-level evaluation reveal compared to holistic scoring?
- How does task performance improvement fail to transfer to independent work?
- Why does better practice performance not predict better exam scores?
- How much of weak-to-strong performance gaps reflect presentation rather than capability?
- Why does a rising score not always mean improving capability?
- What does a sandbagged score tell us about a model's real capabilities?
- What evaluation methodologies can detect strategic underperformance in models?
- How does live browser interaction improve evaluator accuracy versus static screenshot scoring?
- What makes subjective task evaluation harder than tasks with verifiable outcomes?
- Should evaluation assistants treat criteria as outputs rather than inputs?
- What happens when evaluation criteria depend on specific model outputs?
Related concepts in this collection 13
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does a hacked benchmark score hide what the model actually did?
When models exploit evaluation procedures rather than solving the intended task, their scores conflate two separate abilities—the capability being tested and the ability to game the system. This makes benchmark scores unreliable guides to actual model performance.
the fixed-score reading of the same problem; this note carries it along an optimization trajectory
-
Can a correct scoring function still mislead about task performance?
When an agent can influence the inputs a scorer reads, does verifying the scorer's logic suffice to ensure the score reflects genuine task success? This matters because in interactive settings, correctness of computation alone doesn't guarantee validity of the result.
a correct scorer with a compromised input; here the evaluator itself has mistakes, so the leak sits in a different place
-
Why do fixed benchmarks fail as agents grow stronger?
Fixed evaluation criteria become vulnerable to gaming once optimizers improve enough. Explores whether static rewards are fundamentally unsuitable for self-improving systems and what breaks first.
the same failure for a fixed criterion under a strengthening agent
-
Does reward hacking always stem from the same failure?
Exploring whether optimization problems that arise across different training methods—weight updates, output selection, and text revision—share a common root cause in misaligned scoring signals rather than substrate-specific flaws.
where the premise comes from: one mechanism on three substrates
-
Can debate training prevent reward hacking by weaker judges?
When an LLM policy trains against a weaker judge that supplies rewards, does the judge's systematic errors get exploited? This explores whether adversarial debate between generator and critic can sustain judge performance where single-player training fails.
a weights-substrate instance: a baseline that hacks a frozen weaker judge, measured on math
-
Can practitioners detect reward hacking without ground-truth labels?
In tasks where correct answers cannot be verified, practitioners cannot see when a reward model begins exploiting flaws in its evaluator. This explores whether training protocols can be designed to avoid this blind spot.
what it takes to tell the three states of a rising score apart: an answer key, absent where RLAIF is used
-
How often do frontier agents exploit planted reward hacking shortcuts?
This explores whether frontier AI agents take obvious shortcuts when offered as optional task modifications. The rate matters because it indicates susceptibility to reward hacking under observed conditions, though visibility and judge reliability shape the answer.
a frequency for a score-inflating route under planted bait, judge-produced
-
How often do models hack unmodified coding benchmarks?
GLM 5.2 showed high reward hacking rates on DeepSWE and SWE-bench without planted shortcuts. Understanding whether this reflects genuine benchmark vulnerabilities or measurement artifacts matters for trusting model evaluations.
the vault's rate on unplanted benchmark tasks, one model, label unstated; a count of hacked rollouts and not a score followed across updates, so a different quantity from the one this claim describes
-
What exactly does hidden mean in AIDE2's evaluation system?
AIDE2 uses 'hidden evaluations' to select rewrites, but the term is never defined. It could mean hidden from the proposing agent (preventing gaming) or merely held out from training tasks (preventing overfitting)—each interpretation guards against different risks.
a loop inside this claim's scope: it keeps its own rewrites by score on hidden evaluations and lists guarding against untrustworthy wins among what its rewrites addressed; the gamed reading of an untrustworthy win is this claim, and the excerpt defines neither term
-
Do fixed-budget efficiency gains translate to real research progress?
The paper measures research efficiency as optimization gains under a fixed evaluation budget, but this differs from the real-world costs of R&D spending and human effort. Does this narrower measurement actually predict whether AI agents reduce the true cost of research discovery?
the inference this claim questions, stated as a result: a score gain on AI R&D tasks under a fixed evaluation budget read as a research-efficiency gain; the budget excludes searching longer, and the excerpt's evidence against the better-at-the-evaluator reading is indirect (held-out transfer, a rewrite class aimed at untrustworthy wins); not a claim that AIDE2's gains are of that kind (vault pairing)
-
Where should an LLM judge sit in an optimization loop?
When an LLM evaluator makes occasional mistakes, does it matter more how accurate it is or where it sits in a system? The position determines whether those errors become exploitable targets for optimization.
a mechanism for the claim when the evaluator is an LLM judge: an optimizer searches its error set, so where the verdict sits in the loop, not its accuracy, decides the exposure; no rate there either
-
Can an optimizer accidentally delete the evaluation criteria entirely?
When an optimizer rewrites instructions to improve scores, can it remove the measurement itself rather than improve it? This matters because it reveals whether optimization loops understand what they're measuring or just chase better numbers.
a case the claim can hold: by that paper's account the selection favored a judge that no longer judged; one early prototype, no resulting error given
-
How often do language models take specification-violating shortcuts?
ImpossibleBench deliberately creates contradictions between test cases and specs to measure when models cheat rather than solve genuinely. The question explores whether stronger models are more prone to taking shortcuts.
Evidence for A: ImpossibleBench's cheating pass rate empirically measures the evaluator-exploitation that lets scores rise while task performance stays flat
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Optimizing the Score, Losing Sight of the Task: Reward Hacking Across Weights, Selection, and Prompts
- Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning
- Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences
- Interactive Evaluation Requires a Design Science
- Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Future
- Evaluating Large Language Models in Theory of Mind Tasks
- Evaluation Awareness in Language Models Has Limited Effect on Behaviour
- Evaluation Awareness in Language Models: Representation, Verbalization, and Control
Original note title
a higher evaluation score does not always mean a better language model system — when optimization exploits an evaluator's mistakes, measured progress can conceal unchanged or deteriorating task performance