When researchers report how often AI agents get hacked, was it the model that held up — or a hidden filter?

This explores which server-side safety filters, sitting between a tester and the model, may have quietly changed the reported attack-success and defense-gain numbers. The corpus can't give you a list of named backends. What it shows is that the labeling is missing. Attack and defense percentages across the vault are mostly reported without saying whether they were measured behind a server-side filter, so you can't tell if a number reflects the model's own behavior or a filter's Which attack and defense numbers came from filtered backends?. A low attack success rate might mean the model resisted, or that a provider's filter dropped the request before the model saw it.

Other notes explain why this matters and where the same blind spot shows up. A model-level filter judges one output at one moment, while an agent's risk also lives in its memory, retrieved content, tool calls, and reach into its environment Can a model-level filter truly contain an agent with environment access?. A number gathered behind such a filter measures how well the filter caught single outputs, which is a different thing from how safe the agent is. Below that sits the routing layer that decides which model handles a request. It can be manipulated, and safety measures can end up applied to the wrong identity Can attackers manipulate which model handles a request?. So a figure attributed to one model may not have come from the model the paper names.

A zero can also hide what happened. When the Unsafe Action Rate hits zero across defended conditions, it can't separate attacks that were contained from attacks that never took hold. Only measuring compromise at the attacked agent, through the Judgment Bypass Rate, shows the difference Can action-level metrics alone expose contained attacks?. Filtering works the same way: a blocked request and a failed attack both look like zero. A related case is pretraining poisoning. Alignment suppresses jailbreak-style attacks while denial-of-service, context extraction, and belief manipulation survive How much poisoned training data survives safety alignment?. Which attack types get through depends on what sits in front of the model. Some numbers are at least filter-relative in the open. ColluSkill's 96% average success is measured against six named skill scanners Can attackers evade skill scanners by refining individual skills?.

This is one instance of a wider pattern of undisclosed measurement conditions. One benchmark study reports reward-hack rates of 57.2–73% without saying how hacks were labeled How were reward hacks labeled in this benchmark study?. The corpus's proposed fix is to record infrastructure evidence, so an operator can claim that an agent followed the intended evaluation path instead of handing over a bare score Can infrastructure evidence replace terminal scores in benchmark validation?. Until papers report filter status the same way, treat any attack or defense percentage as unverified. The first question to ask of one is what stood between the tester and the model.