Can distance alone rank which substrates resist reward hacking?
Does the amount a method changes a model reliably predict how exposed it is to evaluator errors? The paper tests whether a single distance metric can universally order vulnerability across weights, selection, and prompts.
The abstract states the formal core: "We formalize a distance-dependent upper bound on evaluator disagreement and a capacity ordering for nested policy classes, then show why distance alone cannot establish a universal ranking of vulnerability. An exact finite-output illustration demonstrates how the location of a scoring defect changes the behavior favored by each method." The discussion separates the pieces: "A distance-dependent error envelope gives an upper bound; class inclusion gives a capacity ordering; the geometry of accessible behaviors and the effectiveness of search determine what a particular system actually finds."
Three kinds of statement. The envelope bounds how much evaluator disagreement can matter given how far a policy moves; "how far a policy moves" is the excerpt's only gloss on distance. The ordering says that when one policy class includes another, the larger class has at least the capacity of the smaller. Neither says what a system will find. That depends on where the evaluator's errors sit among the behaviors it can reach and on how well search locates them, which is what the finite-output illustration varies: move the scoring defect and each method favors different behavior. The excerpt does not say the ranking of methods reverses, and it does not give the outputs, the scorer or the methods compared.
Why it is useful (my reading). The tempting shortcut is that the substrate that moves the policy least is the safest. The paper's move is to say a bound is not a forecast and to keep the statements separate so nobody reads one as the other. The point is one a practitioner can use without the formalism: a single number for "how much this method changes the model" does not tell you how exposed the method is to a given evaluator's mistakes, because exposure is set by where the mistake is.
The limit. A finite-output illustration is a constructed case. It supports the claim that no ranking follows from distance alone, and it is not evidence about any deployed system or about which substrate is worse in practice. The excerpt has no formulas, so what "distance" is measured from, and whether the bound is tight, is not stated.
Inquiring lines that read this note 64
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How do prompt injection attacks succeed or fail across defenses?- Does debate's ground-truth protection against hacking transfer to domains without verifiable answers?
- Do synthetic attack traces in papers reflect real adversary behavior?
- Why does search effectiveness determine what method finds despite distance constraints?
- Why do attack success rates alone fail to diagnose system failures?
- How much does attack success depend on tuning to specific scanners versus general robustness?
- What makes a defense mechanism transfer directly rather than just function analogously?
- What properties must defenses preserve to survive substrate differences in persistence and inspectability?
- How does self-preference bias differ from exploitable prompt attack vulnerabilities?
- Do the same attack success rates hold for other backbone models?
- Do multiple frontier models show similar hacking rates on unmodified benchmarks?
- How does frontier model behavior differ between zero-day exploits and infrastructure misconfigurations?
- How does dense task grading compare to honeypot detection for evaluating real capability?
- Can static package analysis find hacks that designers never planted?
- Why do models trained on honeypots still fail on out-of-distribution evaluations?
- Why does treating evaluation as a local output problem miss security risks?
- What makes a win untrustworthy in hidden evaluation environments?
- How does evaluator error position affect which behaviors substrates make vulnerable?
- Can we build reusable evidence that a run stayed within bounds?
- What makes a win untrustworthy and how does AIDE2 avoid spurious optimization?
- How does a single score mix exploitation ability with task capability?
- How should benchmarks balance verifiability against outcome resolution?
- How do non-exploitable vulnerabilities affect benchmark validity?
- How do scoring shortcuts persist across multiple optimization updates?
- How do default fallback scores mask failures in evaluation harnesses?
- What makes a security metric diagnostic rather than outcome-only?
- Does the location of a scoring defect predict which update method will fail?
- How does a ranked default score compete with deliberately optimized outputs?
- Do default score fallbacks in error handling create scoring vulnerabilities?
- Why is error rate alone misleading without strong contestability conditions?
- Why do researchers disagree on open model risks despite same evidence?
- What makes a security boundary evaluation cautious rather than a certification?
- How do frontier models exploit vulnerabilities in their own evaluations?
- How does evaluation environment design become part of the security boundary?
- Can intermediate primitives be scored separately in exploitation benchmarks?
- Why do defense metrics fail without specifying the attacker's position?
- How do missing ground-truth exploits make it harder to identify genuine failures?
- What vulnerabilities have models actually exploited in their own test environments?
- How do evaluation hacks differ from genuine sandbox escapes?
- Why does decoupling evaluation into components make hacking more diagnosable?
- How should capability evaluation environments be secured?
- Can aggregate monitorability metrics miss targeted evasion in specific domains?
- Which reward hacking defenses work across weight updates and output selection?
- Can critics trained in a loop itself become an exploit surface?
- How can hacking stay measurable when ground truth is hidden?
- Do models reward hack at high rates on unmodified benchmarks?
- How does optimization budget interact with a model's vulnerability to reward hacking?
- Is one optimization substrate always safer than another against reward hacking?
- Does reward hacking cause evaluations to overstate model capabilities?
- Can a single hacking vector eliminate the need to anticipate specific exploits in advance?
- How does measurement under loaded conditions differ from measuring real propensity to hack?
- Which reward hacking defenses transfer directly across weights, selection and text?
- Can deterministic guardrails designed for judges transfer to weight-based or text-based reward hacking?
- What reward hacking vulnerabilities emerge from fixed rubrics in RL?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does reward hacking always stem from the same failure?
Exploring whether optimization problems that arise across different training methods—weight updates, output selection, and text revision—share a common root cause in misaligned scoring signals rather than substrate-specific flaws.
the frame this qualifies: same mechanism, no universal ordering of exposure
-
How does the substrate change which behaviors an optimizer can reach?
Different optimization substrates—weights, selection, text—differ in which behaviors they can produce, what information they preserve, and what can be inspected or constrained. Understanding these differences matters because defenses or risk estimates may not transfer between substrates.
the qualitative properties a distance measure leaves out
-
Where should an LLM judge sit in an optimization loop?
When an LLM evaluator makes occasional mistakes, does it matter more how accurate it is or where it sits in a system? The position determines whether those errors become exploitable targets for optimization.
moves the question the same way from how wrong the evaluator is to where its errors sit relative to what the optimizer can reach; the judge note names the loop position, this names the behavior space
-
How should we measure gains from automatic harness evolution?
Harness evolution itself runs a search loop, so reported improvements might come from more search rather than better design. What's the right way to measure whether the harness itself actually improved?
"optimization budgets" is one of the abstract's factors in exposure; that note controls the same variable from the attribution side
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Optimizing the Score, Losing Sight of the Task: Reward Hacking Across Weights, Selection, and Prompts
- Hack-Verifiable Terminal Bench: Evaluating Reward Hacking in Terminal Tasks
- Evaluation Awareness Is Not One Capability: Evidence from Open Language Models
- A Causal Model for Locating and Unlocking Sandbagging in Model Organisms
- Models That Know How Evaluations Are Designed Score Safer
- BAITBENCH: Measuring Agent Reward Hacking with Optional Shortcuts Planted in ML Tasks
- Decomposing and Measuring Evaluation Awareness
- Norm Enforcement for AI Agents: Robustly Shaping Behavior in Multi-Agent Systems
Original note title
distance alone cannot establish a universal ranking of vulnerability to reward hacking across substrates — in the paper's finite-output illustration the location of the scoring defect changes which behavior each method favors