Can static analysis find reward-hacking paths before agents run?
Exploring whether analyzing a task package without running agents can expose exploit-enabling reward-hacking paths. This matters because it could catch vulnerabilities before deployment, without requiring expensive rollouts.
The abstract: "A static, phase-aware taint analysis exposes reward-hacking paths before a run." The conclusion is more specific about the object and the timing: it "discovers exploit-enabling paths in the task package before any agent runs."
Taint analysis comes from software security. It marks data from an untrusted source and follows it to see whether it reaches a sensitive sink. The excerpt does not say what counts as source or sink here. My reading, following the boundary wording in Where does the evaluation boundary actually end in agent benchmarks?: the source is something the agent can affect, the sink is what the outcome procedure reads, and a path between them is an exploit-enabling path. "Phase-aware" then does real work. A path counts only if its steps can occur in an order the lifecycle allows, such as a write that happens before the outcome procedure reads (Can a finite lifecycle model detect reward hacking across benchmarks?).
What a static analysis buys is a property of the benchmark, not of a run. It costs no rollouts and needs no agent. What it cannot say is whether any agent took the path, which is the runtime half's job (Can runtime instrumentation distinguish hacking exposure from actual exploitation?).
The vault already uses taint vocabulary for a different substrate: Can semantic labels on requests prevent malicious propagation through agent networks? labels a request's risk and carries it through a multi-agent workflow at run time. Here the analysis is static and runs over the task package. Same idea borrowed from software security, applied to code paths in one case and to delegation in the other. That comparison is the vault's, not the paper's.
It also touches an open question. A planted honeypot detects only what its designers planted (Can planted honeypots detect hacks that matter most?); an analysis over the package can in principle find paths nobody planted. The excerpt does not say it found any, or how it handles paths that are not visible in the package.
There is a model-side route to the same set of unlisted hacks. Can difference of means vectors discover unknown hacking behaviors? applies a direction across a model's behavior to see what lights up, and the paper it comes from sets up the premise that "anticipating all possible exploits becomes intractable" (Do reward hacking behaviors share a single direction in activation space?). An analysis over a modeled package can be sound and still incomplete against an open-ended exploit space, which is why the filed tension in ops/tensions/ asks what the package model covers. The two look in different places, the package and the model, and neither excerpt reports coverage or any overlap between what they find.
What the excerpt does not give. The language or format of a task package, the source and sink definitions, a false-positive or coverage figure, or any benchmark it was run on.
Inquiring lines that read this note 23
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Do planted honeypot tests reliably measure reward hacking?- Can infrastructure monitoring catch reward hacking that scores cannot detect?
- Can phase-aware static taint analysis scale across different benchmark task types?
- What distinguishes a task that merely exposes a hacking vector from one being actively exploited?
- How do planted honeypots in tasks expose agent exploits differently than task permissions?
- Can static package analysis find hacks that designers never planted?
- How does phase-awareness prevent false positive exploit paths in static analysis?
- What counts as a source and sink in reward-hacking taint analysis?
- What vulnerabilities does runtime instrumentation catch that static analysis misses?
- Do planted test cases reliably detect agent hacking behavior?
- How do planted detectable hacks compare to human inspection of agent traces?
- Can planted hacks within tasks meet the reusability requirement?
- What distinguishes a run that exercises a hacking vector from one that merely exposes it?
- How do evaluation hacks differ from genuine sandbox escapes?
- How do planted errors in honeypot tasks differ from real oversight capabilities?
Related concepts in this collection 6
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can runtime instrumentation distinguish hacking exposure from actual exploitation?
A benchmark task might be vulnerable to hacking without any run actually exploiting it. Can we instrument runtime authority-bearing transitions to tell exposure apart from exercise?
the runtime counterpart; static reports exposure, runtime reports exercise
-
Can a finite lifecycle model detect reward hacking across benchmarks?
Does modeling benchmark runs as typed event lifecycles, checked against task bindings, successfully detect reward-hacking exploits across multiple evaluation tasks? This approach aims to replace task-specific patches with reusable formal detection.
the model whose phases the analysis is aware of
-
Can semantic labels on requests prevent malicious propagation through agent networks?
SafeFlow explores whether attaching structured intent labels to root requests and propagating them through multi-agent collaboration graphs can block malicious information flow by restoring context that task fragmentation strips away.
taint vocabulary on a run-time, semantic substrate; this is the static, code-level use
-
Can planted honeypots detect hacks that matter most?
Hack-Verifiable Terminal Bench embeds known exploits to detect when models take shortcuts. But the original threat was unknown vulnerabilities—hacks the designers never anticipated. Can a planted honeypot measure what it was designed to catch?
a route to hacks nobody planted, if the analysis finds them
-
Can difference of means vectors discover unknown hacking behaviors?
This question asks whether vectors built to detect reward hacking can be used in reverse—applied across a model's behavior to discover hacking patterns nobody labeled in advance, including subtle forms like shortcut deliberation.
the model-side route to unlisted hacks; the package and the model are different places to look, and neither excerpt reports coverage
-
Do reward hacking behaviors share a single direction in activation space?
The note explores whether different ways models exploit evaluation metrics can be detected through a single linear direction in their activations, and whether that direction generalizes across models and settings.
carries the premise that anticipating all possible exploits becomes intractable, the claim the filed tension sets against this analysis
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Hack-Verifiable Terminal Bench: Evaluating Reward Hacking in Terminal Tasks
- BAITBENCH: Measuring Agent Reward Hacking with Optional Shortcuts Planted in ML Tasks
- Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations
- Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation
- BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure
- ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?
- Natural Emergent Misalignment From Reward Hacking In Production RL
- Trust propagation and structural containment in Multi-agent LLM pipelines
Original note title
a static phase-aware taint analysis over the task package discovers exploit-enabling reward-hacking paths before any agent runs