Hacking benchmarks only count a full break-in as success, so should an AI get partial credit for getting halfway?

This explores whether an exploit benchmark could give credit for partial progress, such as reading or writing arbitrary memory or escaping a sandbox, instead of scoring only whether the agent reaches full code execution. The corpus shows the gap clearly. It has no example of anyone closing it. It does hold adjacent ideas that show what closing it would take.

The gap comes from ExploitGym, where success means arbitrary code execution. That endpoint is clear and verifiable, but it treats an agent that got an arbitrary read/write primitive or escaped a sandbox exactly like one that failed on the first step Does arbitrary code execution alone capture exploit progress?. This matters more because exploitation is already the least-measured part of cybersecurity evaluation. Models look strong on vulnerability reproduction, patching and CTFs, while the step where a bug becomes a real attack is barely tracked Do cybersecurity benchmarks actually measure exploitation?. An all-or-nothing score on that step means most of what agents actually do goes unseen.

The nearest machinery comes from reward-hacking research, where the aim is to record what happened during a run instead of trusting the final number. BenchShield models a benchmark run as a finite sequence of typed, reward-relevant events, and checks it against what the task intended Can a finite lifecycle model detect reward hacking across benchmarks?. It also records authority-bearing transitions on the infrastructure side, so it can tell a task that exposes a vector from a run that exercised one Can runtime instrumentation distinguish hacking exposure from actual exploitation?. Those papers use this to detect cheating. The same approach, applied to exploit primitives, would log "got arbitrary read" or "escaped the sandbox" as events and score them separately. That extension is my inference. The papers don't test it. AgentCompass points the same way from the other side: separating benchmark, harness and environment makes agent behavior inspectable, where a scalar score hides it How can we make reward-hacking visible in agent evaluation?.

There are two catches. The first is how partial credit gets built. DRO found that turning rubric scores into dense rewards invites hacking, while using the rubric as a pass/fail gate did not Can rubrics and dense rewards work together without hacking?. A benchmark that scores each primitive independently adds several scoring points an agent could game. It might be safer to report primitives as a checklist of what was reached than to sum them into one number. Related work says how exposed a system is depends on where the evaluator's errors sit and how good the search is, not on a general ranking Can distance alone rank which substrates resist reward hacking?. Each new intermediate check is a new place for a scoring defect.

The second catch is that finer scoring is also a finer dual-use measurement. Exploit capability serves defenders assessing vulnerabilities and lowers the barrier for attackers, and no single measurement separates the two without knowing who has access and under what controls Does measuring exploit capability help or harm defense?. Scoring intermediate primitives would give a more detailed map of how far an agent can get toward an attack. So the technical answer is probably yes, if you record events at the infrastructure level and report them without collapsing them into a reward. Who should see that detailed map is still unsettled.