Can planted honeypots reliably catch reward hacking automatically?

The abstract defines reward hacking as "satisfying a task's checks while violating its intent," and names the measurement problem: detection "typically relies on human inspection or LLM judges, both of which can be unreliable." The hack-verifiable environments (HVE) methodology, which the paper takes from earlier work rather than originates, answers by "embedding detectable hacks into tasks, allowing reward hacks to be identified automatically and reliably." The introduction is more concrete: honeypots are planted "so that hacks are detected automatically rather than judged post hoc."

The move changes what kind of question detection asks. Judging a trace asks whether a sequence of actions looks like cheating, which is an interpretation and inherits the judge's errors. A planted honeypot asks whether one specific, known thing happened, namely whether the agent took the planted route. That is a fact about the run. The experimenter knows the hack because they put it there, so nobody has to recognize it afterward, and that is the sense in which the environment is "hack-verifiable."

Two vault readings, neither of them the paper's. First, a detector that checks for a planted event does not depend on the agent's own account of what it did, so it sits outside the failure the vault documents for trace-reading monitors (Does optimizing against monitors destroy monitoring itself?, Can we detect when models hide their reasoning?). The excerpt says only that judged inspection can be unreliable; it does not say the planted route is immune to obfuscation, and the paper uses the detector for measurement only. Were it also made a training signal, Does iterative training against detected failures prevent actual compliance? predicts selection for avoiding the planted route and not for leaving the hack alone, which is that note's reading of a reliable detector and no run here tests it. Second, the reliability is bought with coverage: detection is reliable for the hack that was planted, which is the subject of Can planted honeypots detect hacks that matter most?.

Notes from other papers in the batch meet this claim from three sides; the comparisons are the vault's, and no excerpt makes them. Can deterministic checks protect LLM judges from failure? uses the same device on a judge inside an optimizer loop, a planted case whose success is the alarm, with the same coverage limit. BaitBench also plants (How often do agents exploit optional shortcuts in benchmarks?) but its headline rate comes out of a judge pipeline (How often do frontier agents exploit planted reward hacking shortcuts?), so the worry recorded here about judged detection applies to that number; a tension is logged for the pair in ops/tensions/. And How were reward hacks labeled in this benchmark study? asks where the label behind a hacking rate comes from on ordinary benchmarks, which a planted hack answers by construction, for the route that was planted only.

What the excerpt does not give. Any measurement of how unreliable human or LLM-judge detection was, what the honeypots in Terminal Bench tasks look like, how many tasks carry one, or any reward-hacking rate.