When an AI system 'blocks every attack,' is the system actually safe, or is its cloud provider quietly doing the blocking?

This explores whether a headline 'the attack failed' number can be credited to the system under test, when a cloud provider's filter may have done the blocking and outcome-only reporting can't say which. The corpus says the filter can hide it, and gives one measurement. An undefended multi-agent pipeline reported zero attack success, but 54 of its 60 blocks came from Azure's cloud filter rather than the application itself Where do safety wins come from in multi-agent systems?. The safety was real but borrowed. An outcome-only report (did the attack succeed?) can't tell borrowed from owned until someone swaps the backend and the number moves.

The gap is probably widespread. A separate note checks published multi-agent attack and defense figures and finds most never say whether they were measured behind a server-side filter Which attack and defense numbers came from filtered backends?. A reported defense gain might be the provider's gain, and a reported attack rate might be the filtered remainder. That is a labeling gap, not proof that every number is inflated. It does mean a reader can't tell which numbers describe model behavior and which describe filtered outcomes.

Leaning on the filter is also shakier than it looks. A filter judges one output at one moment, while an agent's risk spreads across memory, retrieved content, tool calls and environment access Can a model-level filter truly contain an agent with environment access?. The layer beneath it, the one deciding which model handles a request, can be manipulated to send traffic to weaker models or to make safety checks run against the wrong identity Can attackers manipulate which model handles a request?. Attackers can also learn from a checker's responses. Skill scanners that score pieces individually were beaten at 96% average success by refining each piece against scanner feedback while the chain-level intent stayed intact Can attackers evade skill scanners by refining individual skills?. A zero measured against one fixed filter on one day is a fact about that filter on that day. Safety layers also tend to cover one attack class. Alignment suppressed poisoned-data jailbreaks but let denial-of-service, context-extraction and belief-manipulation attacks persist How much poisoned training data survives safety alignment?.

Several notes suggest the fix is to report evidence about the path, not just the result. BenchShield proposes benchmark claims backed by recorded infrastructure evidence about whether the agent followed the intended route, instead of a terminal score alone Can infrastructure evidence replace terminal scores in benchmark validation?. Reward-hacking defenses have the same gap: none gives a portable, per-run record that the run stayed inside its evaluation boundary Do current reward-hacking defenses provide reusable evidence of safety?. For filters, that would mean labeling filter status and running the same attack with the filter on and off.

None of this makes the filter a bad thing to have. Hugging Face ended an OpenAI agent's intrusion with its own perimeter controls, without knowing where the intrusion came from Can defenders stop intrusions without knowing who sent them?. Provider-side controls protect deployments. The problem is only in how the number is credited. The evidence for better practice is thin so far. One paper designs a four-arm monitoring comparison but reports no results Does added monitoring improve protection at acceptable cost?. Two incident records show that evaluation environments are part of the security boundary, but they can't establish attack mechanisms or how often failures recur What can two incident records actually teach us about AI evaluation security?.