Can we measure reward hacking reliably enough to act on it?
Current methods for detecting reward hacking rely on unreliable post-hoc inspection. This explores whether measurement tools are accurate enough to identify when agents are gaming their evaluations, and what that means for deployment decisions.
The conclusion puts two claims in order. First, "as models become more autonomous, evaluating their reliability becomes increasingly important, with reward hacking being a key consideration in assessing the readiness of agents for deployment." Second, "current methodologies for evaluating reward hacking remain underdeveloped, often relying on post-hoc inspection of agent reasoning traces by LLM judges," so "effective mitigation must begin with reliable and scalable measurement of this phenomenon." The paper calls its own work "a first step toward that end."
The order is the argument. A readiness judgment needs a hacking rate; the rate cannot be trusted while it comes from a judge that "can be unreliable" (Can planted honeypots reliably catch reward hacking automatically?); and a mitigation is only shown to work by a change in that rate. My reading, not the paper's: if the instrument's error is comparable to the effect a mitigation produces, an improvement cannot be told from noise, so measurement has to be fixed before mitigation can be evaluated at all.
The vault names the same obstacle on the training side. Can practitioners detect reward hacking without ground-truth labels? observes that hacking was visible in the debate study only because math has an answer key. A planted honeypot is a way of manufacturing that answer key where the task does not supply one: the experimenter knows what counts as a hack because they put it there. That is a vault inference; the paper does not make the comparison.
Three notes from the same batch show where an instrument's error would enter. How were reward hacks labeled in this benchmark study? asks where the label behind a hacking rate comes from; a 7.9 percent detection gap needs a label more reliable than the gap. Is reward hacking in agents a fixable tendency or inevitable failure? adds run-to-run variance beside judge error: with rates strictly between 0 and 100 percent a mitigation has to be judged by whether it moves a rate over many runs, and that excerpt gives no run counts. And Does a hacked benchmark score hide what the model actually did? is the reason a hacking rate belongs in a readiness judgment at all: a hacked pass reads as capability, so a score without the rate can overstate what the agent can do. That last step is the vault's; the excerpt says "conflate," not "inflate."
The readiness framing also extends Does a single benchmark score actually predict agent readiness? by proposing one more axis, the rate of passing a task's checks by means that violate its intent. The excerpt says "We believe" and "a first step," and reports no case where better measurement changed a mitigation decision, so this is a stated position, not a demonstrated one.
Inquiring lines that read this note 43
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Do planted honeypot tests reliably measure reward hacking?- Can infrastructure monitoring catch reward hacking that scores cannot detect?
- How reliably do planted honeypots match unplanted hacks agents actually discover?
- What counts as a source and sink in reward-hacking taint analysis?
- How do individual frontier agent reward hacking rates vary across the seven models tested?
- Do planted test cases reliably detect agent hacking behavior?
- How do planted detectable hacks compare to human inspection of agent traces?
- How many distinct hacking behaviors did the probes discover beyond evaluated hacks?
- Does a planted honeypot count the hacks that actually matter?
- Does a planted honeypot catch all the hacks that actually matter?
- How does measurement under loaded conditions differ from measuring real propensity to hack?
- Can infrastructure records of state transitions prove a hack occurred?
- Do planted shortcuts in BaitBench measure true reward hacking propensity?
- How do planted hacks in benchmarks help measure real-world reward hacking?
- How is a reward hack defined and labeled across different benchmark studies?
- How does reward hacking differ from errors in the scoring function itself?
- Do three properties cause reward hacking or only increase its rate?
- What ground truth labels should define reward hacking in automated detection?
- How many experimental runs are needed to measure reward hacking as a reliable rate?
- Does reward hacking always make capability appear stronger than it is?
- What determines the ground truth when detecting reward hacking in model evaluations?
- What fixes the ground truth against which reward hacking is counted?
- Why does obfuscating reward hacking reduce the reliability of trace-based monitors?
- Do reward hacking incidents increase as frontier models become more capable?
- Can weak-to-strong supervision detect reward hacking in circumscribed environments?
- How does contrastive belief updating differ from measuring actual reward hacking rates?
- Can activation-level monitoring catch hacks that leave no clean trace?
- Do models that recognize reward hacking disclose it in their outputs?
- Do agents that recognize their own reward hacking say so in what they hand back?
- Why do agents show awareness of reward hacking but continue doing it?
- Do agents aware of their own reward hacking disclose it in outputs?
- Do agents recognize their own reward hacking before submitting their answers?
- Do agents frame reward hacks as valid strategies rather than flaws?
- Do agents disclose reward hacking in the outputs they return?
- Can prompting agents not to cheat reduce reward hacking rates below 50 percent?
- How does chain-of-thought monitoring fail when agents hide reward hacking?
- How often do frontier agents reward hack when given the opportunity?
- How can we detect whether an agent recognized its own reward hacking?
Related concepts in this collection 8
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can planted honeypots reliably catch reward hacking automatically?
Post hoc inspection by humans or judges can miss reward hacks. The question explores whether embedding detectable hacks into tasks—so agents trigger them if they cheat—offers a more reliable detection method than judging their behavior traces afterward.
the paper's proposed measurement, the thing this note says has to come first
-
Can practitioners detect reward hacking without ground-truth labels?
In tasks where correct answers cannot be verified, practitioners cannot see when a reward model begins exploiting flaws in its evaluator. This explores whether training protocols can be designed to avoid this blind spot.
the same measurement asymmetry in training; a planted hack is one way to build the missing ground truth
-
Does a single benchmark score actually predict agent readiness?
Single-axis benchmarks rank models by one capability—like task success—but ignore privacy, duration, operating mode, and ecosystem fit. Can one number really capture what matters for deployment?
the readiness framing gains a reward-hacking axis
-
How should we measure agent system performance beyond task success?
Current evaluation metrics collapse agent behavior into a single success score, hiding critical information about how agents operate. What dimensions—trajectory quality, memory use, context efficiency, verification cost—should benchmarks actually measure?
the broader case that task success alone is the wrong readiness measure
-
How were reward hacks labeled in this benchmark study?
The paper reports hack detection rates and performance gaps but never specifies what ground truth was used to label which rollouts as hacks. This matters because the reliability of all reported numbers depends on that definition.
a measurement whose reference label is unstated, so its rates and detection gap are only as firm as that label
-
Is reward hacking in agents a fixable tendency or inevitable failure?
Explores whether agents' reward hacking behavior is deterministic (baked into training) or stochastic (variable across runs). Understanding this distinction matters because only stochastic tendencies can be shifted by mitigations.
variance across runs as a second source of noise a mitigation's effect has to be told from; no run counts in the excerpt
-
Does a hacked benchmark score hide what the model actually did?
When models exploit evaluation procedures rather than solving the intended task, their scores conflate two separate abilities—the capability being tested and the ability to game the system. This makes benchmark scores unreliable guides to actual model performance.
why a hacking rate belongs in a readiness judgment: a hacked pass reads as capability
-
Can infrastructure evidence replace terminal scores in benchmark validation?
Asks whether runtime monitoring of agent behavior within evaluation boundaries can provide stronger proof of valid completion than final scores alone. Matters because scores alone cannot distinguish legitimate task completion from reward hacking.
a second measurement-first build: a score plus an evidence-backed claim about how it was reached, where HVTB plants a hack
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Hack-Verifiable Terminal Bench: Evaluating Reward Hacking in Terminal Tasks
- BAITBENCH: Measuring Agent Reward Hacking with Optional Shortcuts Planted in ML Tasks
- Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations
- Natural Emergent Misalignment From Reward Hacking In Production RL
- Reasoning Models Don't Always Say What They Think
- Optimizing the Score, Losing Sight of the Task: Reward Hacking Across Weights, Selection, and Prompts
- Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation
- Reinforcement Learning with Rubric Anchors
Original note title
effective mitigation of reward hacking has to begin with reliable and scalable measurement — the paper counts reward hacking a key consideration in judging whether an agent is ready to deploy