If two teams test the same AI agent step by step, what would make their scores truly comparable?

This explores what shared tooling and reporting conventions would let different teams run the same multi-step agent evaluation and trust they're comparing like with like. The corpus has no finished standard. It does point to three layers: what you record, what counts as part of the test, and what you disclose alongside a score.

Start with why this is hard. Scoring a whole trajectory instead of a final answer doesn't remove the old problems of comparability, reproducibility, and turning evidence into a judgment. It moves them into a much bigger space Do interactive evaluations actually solve the benchmark comparison problem?. Two runs of the same agent can split apart at step three, so one accuracy number hides most of what differs. The corpus's diagnosis is that the field needs design protocols and shared standards, not just adoption of the interactive format Should interactive evaluation be designed as a unified paradigm?. In practice that means agreeing on how evidence expands from final responses to full interaction sequences, and on how process quality, recoverability, coordination, and robustness get scored How should we evaluate agent behavior beyond final answers?.

The infrastructure idea with the most bite is BenchShield. Benchmark operators record evidence at the infrastructure level, so they can make a verifiable claim that an agent followed the intended evaluation path, instead of publishing only a terminal score Can infrastructure evidence replace terminal scores in benchmark validation?. That changes what reproducible means. Two runs need not produce the same number. What matters is whether each run is a valid completion you can check. This only works if the boundary is drawn widely enough. Agents can change state early and then receive feedback that shapes later actions, so logs, feedback channels, and persistent artifacts all sit on the reward path and belong inside the evaluation boundary Where does the evaluation boundary actually end in agent benchmarks?. A reporting standard would therefore have to pin down and disclose those components, not just the task and the model. Security incident records support the point that evaluation environments are part of the security boundary. Only two preliminary records exist, though, and they don't show how often failures recur or which controls work What can two incident records actually teach us about AI evaluation security?. A standard can treat this as a known risk without yet claiming to know the attack patterns.

For what to report, a neighboring field offers a useful pattern. In post-training, the same reasoning data has different effects depending on the verifier, base model, lineage, optimizer, scaffold, and compute budget. Results only become attributable when those are released together What is the actual reusable unit of reasoning data?. That note is about training data, not evaluation, but the reporting logic carries over. A trajectory-level result probably needs to ship with its scorer, environment, scaffold, and budget, or it can't be reproduced. This is my inference rather than something the corpus states.

The scorer is part of the infrastructure too. An agent-as-judge that gathers its own evidence cut judge shift to 0.27%, against 31% for an LLM judge on complex tasks. Its memory module cascaded errors, though, so agentic judges need error isolation to keep the gain Can agents evaluate AI outputs more reliably than language models?. Measurement is also still patchy. Even for a property like whether errors stay visible and recoverable, only partial instruments exist, and none covers the full picture How can we measure whether AI errors stay visible and recoverable?. Until shared instruments exist, a reproducible report has to say which instruments it used and what they leave out.