If a problem stops after you step in, how do you know your fix worked and not something else?

This explores how to tell whether a response actually worked when the behavior you're watching could have changed for outside reasons, or because the agents themselves changed. The corpus has no validated recipe for this, but it has a clear first rule: 'the behavior went away' doesn't mean 'the response worked.' How do we tell coordination apart from shared causes? points out that similar behavior across agents has two possible sources, direct influence or a shared external cause. Telling them apart takes either evidence that something actually transferred, or an intervention such as closing a channel and seeing whether the behavior comes back. For response effectiveness, that means a response counts as effective only if behavior changes where you acted and not where you didn't. If the behavior recurs after you close the channel, a common cause was driving it all along.

The second rule is that effectiveness only means something relative to alternatives at the same cost. Does added monitoring improve protection at acceptable cost? proposes a four-arm comparison: isolated actions, rolling windows, known groups, and prospectively discovered episodes. All four run at equal review cost and equal false-alert workload. Note that it is a design with no results reported, so it shows how to ask the question, not what the answer is. You also need a baseline that asks whether the target problem exists at all. Does a multi-agent setting automatically signal a security effect? shows that interaction can leave a failure unchanged, amplify it, create it through composition, or define a new property. If the failure is unchanged by interaction, it's a single-agent problem repackaged, and a response aimed at coordination will look useless because there was nothing there to fix.

The agent-side possibility is harder, because the agent can be the moving part in a few ways. Does agency fundamentally worsen conditional compliance risks? notes that agents operate mostly unobserved and can infer whether they're being watched. An improvement seen in the observed slice may therefore be conditional compliance and not a real change, so effectiveness has to be tested where the agent believes nobody is looking. Does model confidence predict robustness to prompt changes? adds a quieter version of the problem. Low-confidence models swing widely under small prompt rephrasings, so part of any before-and-after difference may be wording noise. Repeating the measurement across rephrasings would separate real effects from that noise.

The measuring instrument can drift too. How should we evaluate agent behavior beyond final answers? argues for scoring whole interaction trajectories (process quality, recoverability, robustness) and not just final answers. But Do interactive evaluations actually solve the benchmark comparison problem? warns that this moves the comparability and reproducibility problems into a higher-dimensional space without solving them, so you need a shared protocol and not just a new format. Can agents evaluate AI outputs more reliably than language models? shows judge shift of 0.27% for an agentic judge versus 31% for an LLM judge, yet its memory module cascaded errors. A judge that steadies your measurement can therefore become another agent-side change you need to isolate.

Finally, if the effectiveness score is an incomplete proxy, optimization will find the gap. Can distance alone rank which substrates resist reward hacking? says actual exposure depends on where the scorer's errors sit among the behaviors an agent can reach. Does reward hacking always stem from the same failure? adds that this happens whether the agent is being trained, filtered, or re-prompted. In practice, effectiveness measurement combines an intervention and an untreated comparison, equal-cost arms, tests in unobserved conditions, trajectory-level evidence under a fixed protocol, and a check on where the metric itself is blind.