What causes failures in exploitation benchmarks?

The ExploitGym authors are candid that a failed attempt does not mean what a reader would first assume. Their limitations paragraph lists three routes to failure that have nothing to do with how hard it is to craft an exploit payload: "refusal due to safety alignment, tool misuse, or other underlying causes," and vulnerabilities that turn out to be non-exploitable, "where success is impossible." All of these produce the same observable result as inability: no working exploit.

The consequence is that the benchmark's failure cell is a mixture. A model that refuses is scored the same as a model that tried and could not do it, and both are scored the same as any model on a task nobody could have solved. So a low success rate is best read as a lower bound on capability, not an estimate of it. That matters most in exactly the setting this benchmark serves, where the question behind the number is "how dangerous could this agent be?" A number that can be depressed by a safety refusal understates that danger, and the excerpt gives no way to tell by how much. The paper's results may quantify refusals; this note rests only on the limitations paragraph. A neighboring case of an agent's own tool error sits behind a security number elsewhere: How many GPT-MAS failures came from tool access confusion? reports most of one system's failures as a wrong belief about tool access, in a different benchmark, though that excerpt is cut off before saying what counted as a failure.

This is the mirror image of a failure the vault already tracks. Do autonomous agents report success when actions actually fail? is about false success: the agent says it worked and it did not. Here the risk is false failure: the benchmark says it did not work when the agent was unwilling, mis-operated its tools, or was handed an impossible task. Both are measurement errors in the outcome label, in opposite directions. It also differs from Can language models secretly underperform on safety evaluations?, where the underperformance is strategic; refusal from safety alignment and tool misuse are not strategies, but they hide capability from the evaluator just as effectively.

The fix that follows, in my reading, is attribution: separating refusals, tool errors and impossible instances from real attempts. The excerpt does not say whether the paper does this. That is the kind of analysis that How can we make reward-hacking visible in agent evaluation? argues requires trajectory-level visibility. The non-exploitable-vulnerability case is harder, because it cannot be resolved without ground-truth solutions — see Can scarcity of solutions protect benchmarks from data contamination?.