Could software read a benchmark task's code and automatically work out what counts as cheating, instead of a human spelling it out?

This explores whether the task-specific details a reward-hacking checker like BenchShield needs (the 'task bindings' that say what counts as what in one particular benchmark task) could be worked out by analyzing the task's code automatically, instead of being written by hand. The corpus doesn't answer this directly, but it has the two halves of the puzzle, and they point in a suggestive direction.

First, the problem. BenchShield describes a benchmark run as a finite, typed lifecycle of reward-relevant events and flags deviations from the intended lifecycle, using the same formal object for pre-run static analysis and runtime monitoring (Can a finite lifecycle model detect reward hacking across benchmarks?). But runs are checked against validated task bindings, and the paper never says who writes them, how they're validated, or what one costs. If each task needs its own hand-built binding, the claim that BenchShield is more reusable than task-specific patches is one of degree, not kind (How reusable is BenchShield if task bindings require per-task work?). Whether the bindings can be derived automatically is therefore the question the reusability claim rests on.

Second, the closest evidence that static analysis could do the job. A separate line of work runs a phase-aware taint analysis over the task package itself. It tracks data flowing from things the agent controls to the things that decide the outcome, in phase order, with no agent run and no compute spent on execution (Can static analysis find reward-hacking paths before agents run?). To do that, it has to identify from the code which pieces are agent-controllable sources and which are outcome-procedure sinks. That is much of what a binding seems to encode. So the machinery for reading task-specific structure out of a task package already exists, at least for exploit discovery. Whether that same analysis can be turned around to produce bindings, and how well, isn't shown.

The corpus also has two adjacent ideas for the parts that static analysis might miss. Agent Workflow Memory shows that reusable routines can be induced from past examples by abstracting away example-specific values (Can agents learn reusable sub-task routines from past experience?). A similar move could turn a handful of hand-written bindings into a template for new tasks. Structured reasoning can also verify code behavior without running it, at 93% accuracy in one setting (Can structured reasoning replace code execution for RL rewards?). A model could in principle propose bindings that way. That accuracy was measured on patch equivalence, though, not on binding derivation, so it shows the direction is plausible and nothing more.

The open gap is validation. Even if analysis proposes a binding, someone still has to trust it, and the corpus offers no evidence on how far a derived binding can be trusted, or on how much human checking it would still need. Until a paper reports that, 'without manual effort' is a hypothesis the corpus makes reasonable but does not confirm.