ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?

AI agents are rapidly gaining capabilities that could significantly reshape cybersecurity, making rigorous evaluation urgent. A critical capability is exploitation: turning a vulnerability, which is not yet an attack, into a concrete security impact, such as unauthorized file access or code execution. Exploitation is a particularly challenging task because it requires low-level program reasoning (e.g., about memory layout), runtime adaptation, and sustained progress over long horizons. Meanwhile, it is inherently dual-use, supporting defensive workflows while lowering the barrier for offense. Despite its importance and diagnostic value, exploitation remains under-evaluated. To address this gap, we introduce ExploitGym, a large-scale, diverse, realistic benchmark on the exploitation capabilities of AI agents. Given a program input that triggers a vulnerability, ExploitGym tasks agents with progressively extending it into a working exploit. The benchmark comprises 898 instances sourced from real-world vulnerabilities across three domains, including userspace programs, Google’s V8 JavaScript engine, and the Linux kernel. We vary the security protections applied to each instance, isolating their impact on agent performance.

Introduction. Recent progress in large language models (LLMs) and AI agents has led to rapid improvements in cybersecurity capabilities, making rigorous evaluation increasingly urgent. Prior work has introduced benchmarks for a range of cybersecurity-related tasks, such as vulnerability reproduction [62], patch generation [63], and Capture-the-Flag problem solving [49, 65]. Frontier models now achieve strong performance on many of these benchmarks [2, 38], highlighting the need to better understand and evaluate the boundaries of their cybersecurity capabilities. Exploitation: A Critical Missing Piece in Cybersecurity Evaluation. A crucial yet underexplored capability is vulnerability exploitation. Exploitation is a challenging task that starts from an initial vulnerability (e.g., a few-byte buffer overflow), progressively obtains stronger primitives and privileges (e.g., arbitrary memory reads/writes), and ultimately causes a concrete security impact (e.g., unauthorized file access or code execution).

Discussion / Conclusion. Limitations. First, our tasks do not cover the full space of exploitation targets, such as Windows, iOS, and Android, or applications that run in those environments. Second, we use arbitrary code execution as the success criteria. While this provides a clear and severe measure of impact, it does not capture other meaningful outcomes, such as arbitrary read/write primitives, sandbox escape without code execution, or partial exploit progress. Third, failures may result from refusal due to safety alignment, tool misuse, or other underlying causes unrelated to the complexity of crafting exploit payloads. Failures may also stem from non-exploitable vulnerabilities, where success is impossible. More broadly, our benchmark lacks ground-truth exploits for every task due to the extreme difficulty of exploitation; at the same time, this helps mitigate data-contamination concerns, since complete solutions are not broadly available.