Can distance alone rank which substrates resist reward hacking?
Does the amount a method changes a model reliably predict how exposed it is to evaluator errors? The paper tests whether a single distance metric can universally order vulnerability across weights, selection, and prompts.
The abstract states the formal core: "We formalize a distance-dependent upper bound on evaluator disagreement and a capacity ordering for nested policy classes, then show why distance alone cannot establish a universal ranking of vulnerability. An exact finite-output illustration demonstrates how the location of a scoring defect changes the behavior favored by each method." The discussion separates the pieces: "A distance-dependent error envelope gives an upper bound; class inclusion gives a capacity ordering; the geometry of accessible behaviors and the effectiveness of search determine what a particular system actually finds."
Three kinds of statement. The envelope bounds how much evaluator disagreement can matter given how far a policy moves; "how far a policy moves" is the excerpt's only gloss on distance. The ordering says that when one policy class includes another, the larger class has at least the capacity of the smaller. Neither says what a system will find. That depends on where the evaluator's errors sit among the behaviors it can reach and on how well search locates them, which is what the finite-output illustration varies: move the scoring defect and each method favors different behavior. The excerpt does not say the ranking of methods reverses, and it does not give the outputs, the scorer or the methods compared.
Why it is useful (my reading). The tempting shortcut is that the substrate that moves the policy least is the safest. The paper's move is to say a bound is not a forecast and to keep the statements separate so nobody reads one as the other. The point is one a practitioner can use without the formalism: a single number for "how much this method changes the model" does not tell you how exposed the method is to a given evaluator's mistakes, because exposure is set by where the mistake is.
The limit. A finite-output illustration is a constructed case. It supports the claim that no ranking follows from distance alone, and it is not evidence about any deployed system or about which substrate is worse in practice. The excerpt has no formulas, so what "distance" is measured from, and whether the bound is tight, is not stated.
Inquiring lines that read this note 50
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How prevalent is reward hacking in frontier models?- Does debate's ground-truth protection against hacking transfer to domains without verifiable answers?
- Do multiple frontier models show similar hacking rates on unmodified benchmarks?
- How do scoring shortcuts persist across multiple optimization updates?
- What makes a win untrustworthy in hidden evaluation environments?
- Why does search effectiveness determine what method finds despite distance constraints?
- How does evaluator error position affect which behaviors substrates make vulnerable?
- Can critics trained in a loop itself become an exploit surface?
- How do non-exploitable vulnerabilities affect benchmark validity?
- Is one optimization substrate always safer than another against reward hacking?
- Does the location of a scoring defect predict which update method will fail?
- How do missing ground-truth exploits make it harder to identify genuine failures?
- How does a ranked default score compete with deliberately optimized outputs?
- What makes a win untrustworthy and how does AIDE2 avoid spurious optimization?
- Do default score fallbacks in error handling create scoring vulnerabilities?
- What properties must defenses preserve to survive substrate differences in persistence and inspectability?
- Can deterministic guardrails designed for judges transfer to weight-based or text-based reward hacking?
- How does dense task grading compare to honeypot detection for evaluating real capability?
- Can static package analysis find hacks that designers never planted?
- Can intermediate primitives be scored separately in exploitation benchmarks?
- How does measurement under loaded conditions differ from measuring real propensity to hack?
- How do evaluation hacks differ from genuine sandbox escapes?
- Why does decoupling evaluation into components make hacking more diagnosable?
- Why does treating evaluation as a local output problem miss security risks?
- How do default fallback scores mask failures in evaluation harnesses?
- What makes a security metric diagnostic rather than outcome-only?
- Why do attack success rates alone fail to diagnose system failures?
- What makes a security boundary evaluation cautious rather than a certification?
- Why is error rate alone misleading without strong contestability conditions?
- How can a single instrument measure errors across multiple system layers?
- How do frontier models exploit vulnerabilities in their own evaluations?
- How does evaluation environment design become part of the security boundary?
- What vulnerabilities have models actually exploited in their own test environments?
- How should benchmarks balance verifiability against outcome resolution?
- Can we build reusable evidence that a run stayed within bounds?
- How should memory poisoning success be scored at the validator stage?
- How should response effectiveness be measured when common causes and agent-side changes are both possible?
- Which reward hacking defenses work across weight updates and output selection?
- How can hacking stay measurable when ground truth is hidden?
- Do models reward hack at high rates on unmodified benchmarks?
- How does optimization budget interact with a model's vulnerability to reward hacking?
- Does reward hacking cause evaluations to overstate model capabilities?
- Can a single hacking vector eliminate the need to anticipate specific exploits in advance?
- Which reward hacking defenses transfer directly across weights, selection and text?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does reward hacking always stem from the same failure?
Exploring whether optimization problems that arise across different training methods—weight updates, output selection, and text revision—share a common root cause in misaligned scoring signals rather than substrate-specific flaws.
the frame this qualifies: same mechanism, no universal ordering of exposure
-
How does the substrate change which behaviors an optimizer can reach?
Different optimization substrates—weights, selection, text—differ in which behaviors they can produce, what information they preserve, and what can be inspected or constrained. Understanding these differences matters because defenses or risk estimates may not transfer between substrates.
the qualitative properties a distance measure leaves out
-
Where should an LLM judge sit in an optimization loop?
When an LLM evaluator makes occasional mistakes, does it matter more how accurate it is or where it sits in a system? The position determines whether those errors become exploitable targets for optimization.
moves the question the same way from how wrong the evaluator is to where its errors sit relative to what the optimizer can reach; the judge note names the loop position, this names the behavior space
-
How should we measure gains from automatic harness evolution?
Harness evolution itself runs a search loop, so reported improvements might come from more search rather than better design. What's the right way to measure whether the harness itself actually improved?
"optimization budgets" is one of the abstract's factors in exposure; that note controls the same variable from the attribution side
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Optimizing the Score, Losing Sight of the Task: Reward Hacking Across Weights, Selection, and Prompts
- Hack-Verifiable Terminal Bench: Evaluating Reward Hacking in Terminal Tasks
- A Causal Model for Locating and Unlocking Sandbagging in Model Organisms
- BAITBENCH: Measuring Agent Reward Hacking with Optional Shortcuts Planted in ML Tasks
- Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations
- Agentic Systems as Boosting Weak Reasoning Models
- LLM-as-a-Judge Is Not an Oracle: Why Self-Improving Agents Need Deterministic Guardrails
- LLMs Can Covertly Sandbag on Capability Evaluations Against Chain-of-Thought Monitoring
Original note title
distance alone cannot establish a universal ranking of vulnerability to reward hacking across substrates — in the paper's finite-output illustration the location of the scoring defect changes which behavior each method favors