Before a benchmark-cheating detector can judge a run, someone has to set up each task — so how much setup does that take?

This explores how many task-specific bindings BenchShield needs across benchmarks. The corpus has no count, and the paper it draws on doesn't seem to give one. The nearest thing is a flagged gap: How reusable is BenchShield if task bindings require per-task work? says BenchShield checks every run against 'validated task bindings' but never says who writes them, how they're validated, or what one costs.

A task binding is the reference that BenchShield compares a run against. Can a finite lifecycle model detect reward hacking across benchmarks? describes the system as treating a benchmark run as a finite, typed sequence of reward-relevant events, an 'intended lifecycle'. Reward hacking is then flagged as deviation from that lifecycle, not by matching known exploit patterns. The binding is what ties the generic lifecycle model to one concrete task. The same formal object supports both a static check before the run and instrumentation during it. Whether one binding covers a whole benchmark, a task family, or each individual task is not stated.

That missing number matters because of how BenchShield is pitched. It is sold as more reusable than task-specific defenses, which are essentially one-off patches. The reusability note points out that if bindings turn out to be per-task artifacts, the difference from patches is one of degree, not kind. You'd still be writing something by hand for each task. The claim hinges on the unreported binding count, and on how much of each binding is automatic versus authored.

The other BenchShield note, Can scoped agents reliably judge semantic hacks in runtime analysis?, shows where human or agent effort probably goes. Audit agents get a limited remit, a fixed set of artifacts, and a requirement to cite evidence. Infrastructure records serve as the unchallengeable checks, and the agent's semantic judgment is the arguable step after them. That describes the runtime judging layer, though, not how bindings are produced. The corpus also says that layer's reliability is unquantified.

So the honest answer is that the number isn't available here. The most you can say is that the design depends on bindings existing for the tasks it checks. Their count, who writes them, and what they cost are the open questions to ask of the original paper.