Line of inquiry
Inquiring lines›How do we ensure safe and aligned…›How do adversarial attacks evade A…›this line of inquiry
Do planted shortcuts in benchmarks measure real reward hacking?
A broader line of inquiry — a family of 59 specific questions the research asks around this. Follow one into its inquiring-line page, or move sideways to a related line below.
Questions in this line of inquiry 59
Specific inquiring lines the field asks around this — ordered from the most general framing down to the most specific angle.
- How do planted hacks in benchmarks help measure real-world reward hacking?
- Do unmodified benchmarks expose similar reward hacking vectors without planted shortcuts?
- How do planted detectable hacks compare to human inspection of agent traces?
- Do planted test cases reliably detect agent hacking behavior?
- How do unplanted benchmark hacking rates compare to rates on constructed bait tasks?
- Why do honeypot tasks reveal reward hacking better than standard benchmarks?
- Do honeypot shortcuts in synthetic tasks measure the hacks that actually matter?
- Does a planted honeypot count the hacks that actually matter?
- Does a planted honeypot catch all the hacks that actually matter?
- Can planted hacks within tasks meet the reusability requirement?
- Do agents disclose reward hacking in the outputs they return?
- Do planted shortcuts in BaitBench measure true reward hacking propensity?
- How do planted honeypots distinguish earned passes from coincidental task completion?
- Do agents aware of their own reward hacking disclose it in outputs?
- Do agents recognize their own reward hacking before submitting their answers?
- Can infrastructure monitoring catch reward hacking that scores cannot detect?
- Can prompting agents not to cheat reduce reward hacking rates below 50 percent?
- Does a planted honeypot count the hacks that matter in benchmarks?
- Can static analysis flag exploit-enabling paths in benchmark tasks before agents execute them?
- Can automatic honeypot detection replace human judgment of agent shortcutting?
- How do individual frontier agent reward hacking rates vary across the seven models tested?
- How can we detect whether an agent recognized its own reward hacking?
- Why do agents show awareness of reward hacking but continue doing it?
- How do planted honeypots in tasks expose agent exploits differently than task permissions?
- Do agents that recognize their own reward hacking say so in what they hand back?
- Do planted honeypots look like natural shortcuts to models or obvious tests?
- How does measurement under loaded conditions differ from measuring real propensity to hack?
- What percentage of agent runs successfully manipulated their own scoring evidence?
- Can intermediate primitives be scored separately in exploitation benchmarks?
- What distinguishes a run that exercises a hacking vector from one that merely exposes it?
- Can per-stage results reveal which demand causes agent failure in exploitation?
- How does dense task grading compare to honeypot detection for evaluating real capability?
- Can verifiable environments embed detectable hacks without needing human judgment?
- How do runtime detectors differ from activation-based reward hacking detection?
- What distinguishes a task that merely exposes a hacking vector from one being actively exploited?
- How often do frontier agents reward hack when given the opportunity?
- Why do agents reward hack less on no-signal tasks than on other BaitBench structures?
- How reliably do planted honeypots match unplanted hacks agents actually discover?
- What distinguishes a rate under planted bait from public run rates?
- Can a model truthfully name a shortcut while failing to doubt it?
- How do agents inherit exploit knowledge through shared history?
- Does varying prompt detail about exploits change how much agents reward hack?
- How do detection methods affect measured rates of reward hacking?
- Can planted test cases remain unrecognizable to adaptive optimizers over time?
- Why does decoupling evaluation into components make hacking more diagnosable?
- What specific reward-hacking shortcuts did frontier models find in Terminal Bench?
- Why do models trained on honeypots still fail on out-of-distribution evaluations?
- Why should identifying the spy correlate with output quality?
- What methods could find unplanted hacks that benchmark designers missed?
- How do evaluation hacks differ from genuine sandbox escapes?
- How visible is the optional shortcut to the agent during evaluation?
- How visible or planted is the shortcut when measuring scheming propensity in stress tests?
- Can static package analysis find hacks that designers never planted?
- What unnamed exploits do models discover in training environments?
- How do planted errors in honeypot tasks differ from real oversight capabilities?
- Why does search effectiveness determine what method finds despite distance constraints?
- What counts as a source and sink in reward-hacking taint analysis?
- What makes exploitation a missing piece in cybersecurity benchmarks?
- How many distinct hacking behaviors did the probes discover beyond evaluated hacks?