What blocks rigorous security evaluation of multi-agent systems?

The abstract reports an audit of "44 evaluation and benchmark works" and lists what it finds open: "isolating interaction effects, designing comparable and diagnostic metrics, supporting reuse across MAS designs, and evaluating open-system operation." The conclusion adds the standard the field is measured against, whether "evaluations verify the resulting system-level effect." The excerpt does not say how many of the 44 fall short on each challenge, and does not define "open-system operation."

The first two challenges already have instances in the vault, found without reference to the SoK. Isolating interaction effects is what How does agent architecture affect web security vulnerabilities? does: hold the task, tools and browser fixed and vary only the architecture. That is the design needed to tell an amplified failure from an unchanged one (see Does a multi-agent setting automatically signal a security effect?). The collusion work intervenes on a different piece of the interaction. Does peer behavior actually cause collusion between agents? sets what the peer does, and Does limiting interaction history actually prevent agent collusion? cuts what agents can see of earlier exchanges. Both act on the interaction itself and not on the architecture, though neither excerpt says what was manipulated or by how much. Diagnostic metrics is what a bare attack success rate is not. Where do safety wins come from in multi-agent systems? shows a favorable zero whose cause the number cannot name, and How many GPT-MAS failures came from tool access confusion? shows failures counted as attacks that were not. Each is a case of a number that verifies an outcome without verifying the system-level effect that produced it. Three notes from other papers sit on the same challenge. Can action-level metrics alone expose contained attacks? is a metric added because the action-level rate had reached its floor, though the excerpt does not say how compromise at the agent is scored. What behaviors hide behind a zero crossing rate? is a zero that fits four different behaviors. Can a correct outcome hide protocol violations in multi-agent systems? is a correct verdict over a skipped check. None of these is shown to be among the 44 audited works.

The other two are harder to check against the vault. Reuse across designs would ask whether a benchmark built around one topology survives a change of it. Open-system operation, read plainly, concerns systems whose participants are not fixed in advance, but the excerpt gives no definition, so that gloss is only a reading.

The counterargument is that an audit of what is missing says little about what is best. The four gaps describe the shortfall, not a ranking of the 44 works.

What the excerpt does not give. The per-challenge counts, the named benchmarks, and the definitions of "diagnostic" and "open-system."