When an AI is graded by a script, how do you spot cheating routes that really work, not just ones that look possible?

This explores how tracking the order of stages in a benchmark run keeps a static scanner from crying wolf about reward hacking. The corpus describes the design but reports no false-positive rates, so what follows is the logic of the approach rather than a measured result. The core idea is taint analysis: follow a piece of data from somewhere an agent can touch it (a source) to somewhere that decides the score (a sink). The paper's version is phase-ordered, so it only counts flows that can actually happen in sequence, from agent-controllable sources to outcome-procedure sinks, and it works from the task package alone, before any agent runs Can static analysis find reward-hacking paths before agents run?.

A phase-blind scanner would flag any path that connects an agent-writable location to a scoring routine. But a path that exists on paper can be closed in practice. The agent may only get write access in a phase that ends before the sink reads the data, or the sink may read from a place the agent can't reach during that phase. Asking when the agent can act, and not only what it can touch, drops those paths and leaves flows that are reachable in order.

BenchShield gives 'phase' a formal home. It models a benchmark run as a finite lifecycle of typed, reward-relevant events, checked against what the task says should happen. The same model supports both the pre-run static analysis and runtime instrumentation Can a finite lifecycle model detect reward hacking across benchmarks?. Detection then means deviation from the intended lifecycle, not matching suspicious-looking patterns. Pattern matching is where false alarms tend to come from, because a file write can look like tampering without being one.

There is a wider lesson in this. Checks that judge one action at a time can't express rules that depend on what came before Can stateless checks ever catch sequence-level constraint violations?. A phase-blind scan has the same weakness. It looks at each flow in isolation, while an exploit's validity depends on where the flow sits in the run's history. Phase-awareness adds that missing state.

Phase-awareness narrows the alarms but doesn't settle the question. Static analysis can only say a task exposes a hacking vector. Runtime instrumentation of authority-bearing transitions then tells a run that merely had the vector apart from one that used it, so an exposed task doesn't make every score suspect Can runtime instrumentation distinguish hacking exposure from actual exploitation?. Together they let operators claim a run followed the intended path, based on recorded evidence and not just the final score Can infrastructure evidence replace terminal scores in benchmark validation?.