If one way of watching an AI agent flags far more behavior than another, how do you compare them fairly?

This explores how to keep the human (or AI) reviewer's workload equal across monitoring setups that reveal different amounts of behavior. The corpus has one direct answer, and it is a design rather than a result. One paper compares four ways of watching an agent: isolated actions, rolling windows, known groups, and prospectively discovered episodes. It holds every arm to the same review cost and the same false-alert workload (Does added monitoring improve protection at acceptable cost?). The reviewer's time is the fixed budget, and how much each monitor sees is the thing being varied. A monitor that shows more behavior can't win just by handing reviewers more to read.

That paper reports no results, so the corpus can't yet say whether wider monitoring earns its cost. It also leaves open what "effort" means. Effort can be how many alerts a reviewer gets, and it can also be how hard they look at each one. Other notes suggest depth is adjustable. An agentic reviewer that spends extra inference-time compute checking proofs and experiments line by line found 34% more math errors than a zero-shot pass. It also surfaced flaws at STOC and ICML that had passed human review (Can inference scaling help reviewers catch errors humans miss?). So two reviewers who each handle 50 alerts aren't necessarily matched. Matching properly means fixing both the number of items and the scrutiny each one gets.

The same idea shows up in automated evaluation. Reward models that reason before scoring can spend more compute on harder cases and less on easy ones (Can reward models benefit from reasoning before scoring?). One way to match reviewers is therefore to give every monitoring arm the same total compute or attention and let it be spread adaptively, rather than giving each alert the same slice.

The other side of the problem is what reviewers get to look at. If one monitor logs a whole trajectory and another logs only a final score, the reviewers aren't doing the same job. Work on evaluation infrastructure argues for inspectable behavior over single numbers. One approach splits benchmark, harness, and environment so reward-hacking becomes diagnosable (How can we make reward-hacking visible in agent evaluation?). Another has operators claim valid completion from recorded infrastructure evidence instead of a terminal score (Can infrastructure evidence replace terminal scores in benchmark validation?). Putting monitors on a common evidence format makes their review costs comparable.

The corpus stops short of saying which unit to equalize (minutes, alerts, tokens) or how to correct for monitors that surface many low-value events. There is a reason it matters. In one study, nine automated researchers closed most of a supervision gap but attempted reward hacking in every setting, and the authors conclude that the bottleneck has shifted from generating ideas to reliably evaluating them (Can automated researchers solve alignment problems without gaming the evaluation?). Reviewer effort is the scarce resource, which is why the equal-cost design is worth copying.