Does planting honeypots in real coding tasks detect actual agent hacking?
Hack-Verifiable Terminal Bench moves honeypot detection from games to real-world coding tasks. But does a constructed shortcut measure the hacks agents actually find when deployed, or only how they respond to planted opportunities?
The paper's contribution is a relocation, not a new detector. Its introduction says "prior work instantiated HVE on game-like environments," while "HVTB targets real-world coding tasks, the setting in which agents are actually deployed." The host is Terminal Bench, which the abstract calls "a leading benchmark of real-world terminal and coding tasks." Hack-Verifiable Terminal Bench (HVTB) is that benchmark with honeypots planted in it, used to "measure reward-hacking rates across frontier models" (Can planted honeypots reliably catch reward hacking automatically?).
Why the setting matters: a measurement shown to work only in game-like environments leaves open whether the behavior it counts resembles what agents do when deployed. Moving to coding tasks answers that for the tasks, though not entirely for the hack. The tasks are real, but the honeypot is constructed by the authors, so HVTB measures behavior on real work with a manufactured opportunity, not the rate at which agents find shortcuts nobody planted. That distinction is the subject of Can planted honeypots detect hacks that matter most?.
BaitBench moves the same planted-hack idea into a different family of tasks: three synthetic tabular ML tasks, with the plant a shortcut in the data or modeling task that a hidden test set exposes (How often do agents exploit optional shortcuts in benchmarks?). Side by side, the plant can sit in a terminal environment or in a modeling task, and each design leaves open whether what was planted resembles what agents meet unplanted. That pairing is the vault's, not either excerpt's.
The paper also releases "all environments and agent traces." My reading, not the paper's: released traces would let others compare the automatic detections against what a human or LLM judge says about the same runs, a check the excerpt does not itself report.
HVTB is one of two modifications of the Terminal Bench family in the vault. Can dense subtask grading reveal agent progress on ultra-long tasks? adds resolution to what counts as capability; HVTB adds a probe for whether a pass was earned. Both start from the fact that a single pass rate compresses more than one thing.
What the excerpt does not give. The number of tasks, which frontier models were run, what the hacks are, any hacking rate, or the prompt conditions.
Inquiring lines that read this note 22
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Do planted honeypot tests reliably measure reward hacking?- What distinguishes a task that merely exposes a hacking vector from one being actively exploited?
- How do planted honeypots in tasks expose agent exploits differently than task permissions?
- How do unplanted benchmark hacking rates compare to rates on constructed bait tasks?
- Can automatic honeypot detection replace human judgment of agent shortcutting?
- How reliably do planted honeypots match unplanted hacks agents actually discover?
- How does dense task grading compare to honeypot detection for evaluating real capability?
- Can static package analysis find hacks that designers never planted?
- Do planted test cases reliably detect agent hacking behavior?
- How do planted detectable hacks compare to human inspection of agent traces?
- Can planted test cases reliably trigger alarms before real harm occurs?
- How do planted honeypots distinguish earned passes from coincidental task completion?
- How many distinct hacking behaviors did the probes discover beyond evaluated hacks?
- Does a planted honeypot count the hacks that matter in benchmarks?
- Do planted honeypots look like natural shortcuts to models or obvious tests?
- Why do honeypot tasks reveal reward hacking better than standard benchmarks?
- Can planted hacks within tasks meet the reusability requirement?
- Does a planted honeypot count the hacks that actually matter?
- Does a planted honeypot catch all the hacks that actually matter?
- Can verifiable environments embed detectable hacks without needing human judgment?
- How do planted errors in honeypot tasks differ from real oversight capabilities?
- Do honeypot shortcuts in synthetic tasks measure the hacks that actually matter?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can planted honeypots reliably catch reward hacking automatically?
Post hoc inspection by humans or judges can miss reward hacks. The question explores whether embedding detectable hacks into tasks—so agents trigger them if they cheat—offers a more reliable detection method than judging their behavior traces afterward.
the detection premise this note relocates to real tasks
-
Can dense subtask grading reveal agent progress on ultra-long tasks?
When agent tasks stretch to hours and hundreds of episodes, does breaking them into fine-grained graded subtasks expose meaningful progress that binary pass-fail scoring completely erases? This matters because most agents fail the final outcome anyway.
a different modification of the same benchmark family: scoring resolution rather than an integrity probe
-
Can planted honeypots detect hacks that matter most?
Hack-Verifiable Terminal Bench embeds known exploits to detect when models take shortcuts. But the original threat was unknown vulnerabilities—hacks the designers never anticipated. Can a planted honeypot measure what it was designed to catch?
what remains constructed about a benchmark whose tasks are real
-
Which security protections actually slow down agent exploits?
ExploitGym varies defenses across 898 real-world instances to isolate how each protection affects agent performance. Understanding which defenses matter most to agents versus humans is critical for defenders.
another case where the environment is edited on purpose so behavior becomes attributable
-
How often do agents exploit optional shortcuts in benchmarks?
BaitBench embeds exploitable shortcuts into synthetic tasks to measure whether agents choose to hack public scores rather than solve tasks honestly. This reveals what fraction of agents prioritize inflated metrics over robust solutions.
the same planted-hack idea in synthetic tabular ML tasks, exposed by a hidden test set; the plant moves between task families as it moved here from games to terminal tasks
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Hack-Verifiable Terminal Bench: Evaluating Reward Hacking in Terminal Tasks
- BAITBENCH: Measuring Agent Reward Hacking with Optional Shortcuts Planted in ML Tasks
- Natural Emergent Misalignment From Reward Hacking In Production RL
- Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations
- Natural Emergent Misalignment From Reward Hacking In Production Rl
- Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation
- Reasoning Models Don't Always Say What They Think
- ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?
Original note title
Hack-Verifiable Terminal Bench carries planted honeypots from game-like environments to real-world terminal and coding tasks — the setting where agents are actually deployed