Does arbitrary code execution alone capture exploit progress?

ExploitGym scores success as arbitrary code execution. The authors defend that choice and name its price in the same breath: it "provides a clear and severe measure of impact," but "does not capture other meaningful outcomes, such as arbitrary read/write primitives, sandbox escape without code execution, or partial exploit progress."

There is a mismatch here that is worth stating plainly, because it sits inside the design. The task is built as a staircase. The paper describes exploitation as progressively obtaining stronger primitives and privileges, and the benchmark asks agents to progressively extend a crashing input into a working exploit (see Why does exploitation test multiple reasoning demands at once?). But the scoring looks only at the top step. An agent that reaches arbitrary memory read/write and stops one rung short scores the same as an agent that never got the input to do anything, even though the first has achieved most of what a defender would worry about.

The reason for choosing the top step is a real virtue. A criterion of "did code execute or not" is checkable without judgment, and that is the property Should reasoning benchmarks score final answers or reasoning traces? praises: verify the outcome, not the story about it. The trade is the familiar one between verifiability and resolution. The more unambiguous the end-state test, the coarser it is about everything short of the end state.

This ties the note to the vault's broader push away from single-number outcomes. How should we measure agent system performance beyond task success? argues the final-success number hides most of what matters about how an agent behaved; an intermediate-primitive score would be one concrete way to recover some of it here. The excerpt does not say whether ExploitGym records partial progress in any form, so I am not claiming it is unavailable, only that the headline criterion does not score it.

The same endpoint-only limit has a defender-side counterpart. Can action-level metrics alone expose contained attacks? found an action-level 0 percent that looks identical whether an attack never took hold or was contained, and added a second reading taken upstream of the final action. By analogy, and the mapping is the vault's, not either paper's, an exploitation score read only at code execution treats an attempt that stalled at the start and one that stopped a rung short as the same; a reading at an intermediate primitive would play the role the Judgment Bypass Rate plays there. The ExploitGym excerpt reports no such metric.