Which security protections actually slow down agent exploits?

Most agent benchmarks treat the environment as a fixed backdrop and the agent as the only thing that varies. ExploitGym makes a different design choice: it "vary[ies] the security protections applied to each instance, isolating their impact on agent performance." The protections on a target are a property of the environment, and the benchmark turns that property into something it can manipulate deliberately. The 898 instances are drawn from real-world vulnerabilities across userspace programs, Google's V8 JavaScript engine and the Linux kernel, so the manipulation happens across realistic and varied targets rather than one toy setup.

The methodological payoff is attribution. A single pass rate for an agent on a fixed set of hardened targets mixes together how good the agent is and how hard the defenses are. If protections are varied while the underlying vulnerability is held constant, the difference in outcome is attributable to that protection. That is how I read "vary the security protections applied to each instance," but the excerpt does not spell out the design, so treat the attribution logic as the intent of the design rather than a verified property of it. That gives the benchmark a second kind of question to answer: not only "how capable is the agent?" but "which defenses actually raise the cost of exploitation for an agent?" — a question of direct interest to defenders.

This is the same structural idea as How can we make reward-hacking visible in agent evaluation?, used for a different purpose. There, separating the environment component makes trajectories analyzable and results reproducible. Here, the environment component is separated so it can be varied as an experimental factor. It also fits the view in What blocks scaling from language models to autonomous agents? that environment properties are a first-class axis, not incidental setup.

How does agent architecture affect web security vulnerabilities? makes the same move one level up: task, tools and browser are held fixed and only the agent architecture varies. Its own caveat carries over. Isolating one factor still leaves choices inside it unfixed (how the multi-agent version divides the work), which is one more reason to read the ExploitGym design as intent until the paper's setup is in view. Can independent scenario factors isolate what drives scheming? applies the same isolation to the agent's own conditions, goals, oversight and pressure, in scenarios; its excerpt does not say the factors are fully crossed.

Open, and not answerable from the excerpt: which protections change agent performance, by how much, and whether any that stop human exploit developers stop agents in the same way. The abstract announces the manipulation but the excerpt carries none of the findings. Any post citing this design should cite it as a design, not as evidence about defenses.