A benchmark checks each AI run against 'task bindings' — but who writes those, and how do we know they're right?

This explores who writes and signs off on the 'task bindings' that BenchShield checks benchmark runs against, and how anyone could tell the bindings themselves are correct. The corpus's direct answer is that nobody says. The paper checks each run against validated task bindings but never explains who authors them, how they're validated, or what one costs to produce How reusable is BenchShield if task bindings require per-task work?. This gap affects the paper's headline pitch. BenchShield presents itself as more reusable than task-specific defenses. If a person has to build a binding for every task, the difference from a per-task patch is one of degree, not kind.

The binding is the thing everything else rests on. BenchShield lets a benchmark operator claim that an agent followed the intended evaluation path, backed by recorded infrastructure evidence instead of just a final score Can infrastructure evidence replace terminal scores in benchmark validation?. It spots exploits as deviation from a finite lifecycle of reward-relevant events, rather than by matching known hacks Can a finite lifecycle model detect reward hacking across benchmarks?. But 'deviation from the intended lifecycle' only means something if the binding describing that lifecycle is right. If the binding is wrong, the checker will confidently flag honest runs or pass exploited ones.

The rest of the corpus doesn't answer the question, but it shows why the obvious answers fall short. Checking the checker is not enough. A scoring function can compute correctly and still report a misleading result if the agent altered its inputs or their provenance outside the intended path Can a correct scoring function still mislead about task performance?. So validating a binding's mechanics wouldn't show it captures the right task. Having a panel of validators vouch for it wouldn't settle things either. A quorum of validators can follow the protocol, sign the right messages, and reach consensus while endorsing something invalid Can a quorum of honest validators certify an invalid transition?. Agreement is guaranteed by the protocol, but semantic correctness holds only statistically, depending on how validators behave Can validator consensus guarantee both agreement and semantic correctness?. Reviewers all signing off would show the process ran, not that the binding is right.

Automation is one way out, though the corpus doesn't say BenchShield takes it. Other work synthesizes formal verifiers, including provably correct Lean and z3 checkers, directly from prose policy documents Can we automatically generate formal verifiers from policy text?. Bindings could in principle be generated from a task description. That would move the trust question to the translation step, and someone would still have to validate the translation. The same problem shows up one level up. A separate note observes that when agents act across organizations, no owner is named for the rules governing them, and operators, organizations, regulators, and standards bodies each hold different, possibly conflicting pieces Who enforces invariants when agents cross organizational boundaries?.

The pattern across these notes is that every layer of checking rests on a specification someone wrote. BenchShield is only as trustworthy as its bindings, and its bindings have no named author or audit trail. That makes it an open question worth pressing on, not a settled answer.