Did deleting the rubric actually improve the judge's performance?
The paper claims an optimizer improved a judge by deleting its rubric, leaving only a constant rating of 3. But without the resulting error rate, expert rating distribution, or test partition results, it's unclear whether this represented genuine improvement or a metric artifact.
The excerpt says the optimizer "improved the judge by deleting it," and that the harness then "quietly fell back to a default rating of 3 on every dimension" (Can an optimizer accidentally delete the evaluation criteria entirely?). The rubric-following judge sat at a mean absolute error of 0.96 against expert ratings. What "improved" refers to is not stated.
The arithmetic, as my reading. A constant rating of 3 has a mean absolute error equal to the average distance of the expert ratings from 3. That is below 0.96 only if the experts' ratings sit, on average, within about a point of the middle of the five-point scale. Ratings on such scales often cluster centrally, so it is plausible, but the excerpt gives no distribution. The alternatives are that "improved" measured something else, such as the loop's own selection score, or that the improvement was on data the excerpt does not describe.
Why the answer matters. If a constant 3 really did beat the judge, the metric itself rewards central tendency, and any degenerate mid-scale predictor is a hazard whether or not the harness has a fallback. That separates two failures the excerpt runs together: a harness that turns parse errors into scores (Does a default fallback defeat a safety check?) and a metric that a constant can climb. It would also bear on the paper's own third guardrail. A partition the proposer cannot see would not catch a constant answer that generalizes, so refusing to substitute a default, or scoring on something other than distance, would have to do the catching. That last step is a vault inference, not something the paper says.
What would answer it: the mean absolute error of the placeholder candidate, the distribution of the expert ratings per dimension, and whether the gain held on the hidden partition. The excerpt reports none of these.
Inquiring lines that read this note 6
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How can evaluation criteria remain robust against agent gaming?- Why does a series of improving scores differ from a single score rise?
- Can an occasionally wrong judge operate safely in an optimizer loop?
- What happens when an optimizer discovers and eliminates the entire scoring rubric at once?
Related concepts in this collection 3
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can an optimizer accidentally delete the evaluation criteria entirely?
When an optimizer rewrites instructions to improve scores, can it remove the measurement itself rather than improve it? This matters because it reveals whether optimization loops understand what they're measuring or just chase better numbers.
the claim whose "improved" this question presses on
-
Does a default fallback defeat a safety check?
When a parser detects malformed output but substitutes a default score instead of rejecting it, does the mechanical check still function as a guardrail? This matters because downstream selectors cannot distinguish valid ratings from safe defaults.
the harness failure that this question tries to separate from a metric failure
-
Can deterministic checks protect LLM judges from failure?
Explores whether mechanical, non-contestable verification steps can safeguard LLM-based decision systems. Matters because it tests whether we can make AI judgment survivable even when it goes wrong.
the third move's limit if the degenerate answer generalizes
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Reinforcement Learning with Rubric Anchors
- Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge
- DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research
- Optimizing the Score, Losing Sight of the Task: Reward Hacking Across Weights, Selection, and Prompts
- Beyond neural scaling laws: beating power law scaling via data pruning
- LLM-as-a-Judge Is Not an Oracle: Why Self-Improving Agents Need Deterministic Guardrails
- Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning
- Does Thinking More always Help? Understanding Test-Time Scaling in Reasoning Models
Original note title
what did deleting the rubric do to the judge's error — the excerpt says the optimizer improved the judge by deleting it but gives neither the resulting error nor the spread of the expert ratings