SYNTHESIS NOTE
Topics›Evaluations›this note

Do current reward-hacking defenses provide reusable evidence of safety?

Existing defenses against reward hacking—task-specific patches, prompt instructions, and post-hoc detectors—may work in practice, but do they leave behind portable, per-run evidence that an evaluation stayed within its intended boundary?

Synthesis note · 2026-09-24 · sourced from Evaluations

The abstract states the gap in two sentences: "Existing defenses rely largely on task-specific patches, prompt instructions, or post-hoc detectors. They do not provide reusable evidence that a concrete run remained within its intended evaluation boundary." The excerpt gives no survey or citation behind "rely largely"; it is the paper's positioning statement.

The sentence packs a taxonomy and a criterion. The taxonomy: a patch fixes one task after a hack is found, an instruction tells the agent not to, a detector looks after the run. The criterion has three parts: the evidence must be reusable (carry across tasks, not be written for one), about a concrete run, and evidence that the run stayed within its boundary. My reading, not the paper's: the third part is a positive claim. A post-hoc detector that flags nothing supports "no hack was detected," which is weaker than "this run stayed inside the boundary," and the paper's wording asks for the second.

The vault has evidence on two of the three types, but it bears on whether the defense works, not on what it leaves behind. Instructions: the mean cheating rate stayed above half when agents were prompted not to (Can prompting agents not to cheat actually stop them?), though without a per-condition split. Detectors: judged inspection "can be unreliable" (Can planted honeypots reliably catch reward hacking automatically?). A detector read off activations is measured against LLM monitors in How do cheap vector detectors compare to expensive LLM monitors?, again a result on whether the detector works and not on what it leaves behind. On patches, the reported hacks are of "previously unknown vulnerabilities" (Do frontier models exploit unknown vulnerabilities in evaluations?), which a patch written for a known hole would not cover. That last link is the vault's inference. So the paper's diagnosis and the vault's findings converge, but the paper's complaint is one level up from theirs: even a defense that works would not hand the operator a reusable record.

A planted hack is a fourth thing the sentence does not name. It is built per task, so whether it meets "reusable" is a vault question the excerpt does not settle.

Inquiring lines that read this note 84

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

How can evaluation systems stay reliable and resistant to AI-driven gaming? Do planted honeypots reliably distinguish genuine task completion from reward hacking? How do prompt injection attacks succeed or fail across defenses? What prevalence and forms of reward hacking occur in model evaluation? What unintended behaviors emerge in frontier AI models? Does multi-agent architecture create security vulnerabilities single-agent defenses can't address? What evidence best independently verifies agent behavior for safety? Can sequences of locally safe agent actions violate system-level constraints? How should agent identity and authorization work across organizational boundaries? How should evaluation environments be designed as security boundaries? How should persistent agent-authored code be governed and shared? How should evaluation expand to capture deployment realities? Do safety evaluations reliably measure real deployed AI safety risks? What happens to the meaning and authority of expertise when AI generates knowledge? Do evolved harness optimizations generalize beyond their source benchmarks?

Related concepts in this collection 8

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
16 direct connections · 88 in 2-hop network ·medium cluster Open in graph ↗

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Original note title

existing defenses against reward hacking rely largely on task-specific patches, prompt instructions or post-hoc detectors — and do not provide reusable evidence that a concrete run stayed within its evaluation boundary