Before worrying about a new AI model, ask how much easier it makes an attack than what attackers already have.
What framework measures marginal offense risk against existing attack technology?
This explores which framework asks how much extra danger a new technology adds beyond what attackers can already do, and what the library says about measuring it.
This explores which framework asks how much extra danger a new technology adds beyond what attackers can already do. The library's answer is the marginal-risk framework for open foundation models. It reframes the policy question. Instead of asking whether an open model can be misused, which is almost always yes, it asks how much easier the model makes misuse than the technology attackers already have. The library's note on it is blunt: across vectors like cyberattacks and bioweapons, the research isn't there yet to measure that marginal effect. Can we measure how much risk open models actually add? The framework exists, but the measurements to fill it in mostly don't.
Cybersecurity shows why. To say a model adds offensive risk, you have to measure its ability to exploit, meaning turning a vulnerability into a working attack. Benchmarks are strong on the neighbouring steps, such as reproducing vulnerabilities, writing patches and solving CTF puzzles, but exploitation is largely unmeasured. Do cybersecurity benchmarks actually measure exploitation? The offense side of the marginal-risk number is the part with the least data.
Even a good exploitation score wouldn't settle the question. The same capability supports defensive vulnerability assessment and lowers the barrier for attackers, and no single measurement separates the two without knowing who has access and under what controls. Does measuring exploit capability help or harm defense? So marginal risk can't be read off a benchmark. It depends on who gets the capability and what safeguards surround it.
The baseline of existing attack technology also keeps moving, especially for AI agents. ColluSkill reaches 96% average attack success against six skill scanners by using scanner feedback to blunt each skill's suspiciousness while the chain-level plan stays intact. Can attackers evade skill scanners by refining individual skills? SafeFlow shows harmful intent split into subtasks that each look benign. Can task decomposition hide harmful intent across agents? A new model's added risk has to be judged against attacks like these, which checks that inspect one piece at a time already miss.
Two neighbouring ideas help if you want to compare attacks and defenses rather than just count risk. The A-I-R framework adds adversary position as a missing axis for comparing multi-agent attacks, so results from different studies stop being apples-to-oranges. Can adversary position unify fragmented multi-agent attack models? And on the defense side, one argument is that the unit to defend should be a revisable set of linked actions across agents, not a single execution. Should defence units span multiple executions and agents?
Sources 7 notes
A marginal-risk framework shows that the policy question should compare open models to pre-existing technology, not assess them in absolute terms. Across vectors like cyberattacks and bioweapons, research is insufficient to measure this marginal effect.
ExploitGym shows that while frontier models excel at vulnerability reproduction, patch generation, and CTF tasks, exploitation—the step where a vulnerability becomes a real attack—remains largely unmeasured in the benchmark literature.
ExploitGym demonstrates that exploit generation supports both defensive vulnerability assessment and lowering barriers to offensive attacks simultaneously. No single measurement distinguishes between these outcomes without knowing who has access and under what controls.
ColluSkill combines chain planning with scanner-feedback refinement to reach 96% average attack success. The approach works because scanners score skills individually, allowing feedback to reduce suspicion per skill while chain-level semantics remain intact.
SafeFlow demonstrates that multi-agent systems' core strength—splitting tasks and specializing roles—creates a safety blind spot where malicious intent can be distributed across steps that each appear benign individually, with harm emerging only in composition.
Show all 7 sources
The SoK's A-I-R framework adds adversary position as a coordinate alongside interaction interface and system-level risk. Existing vault notes vary adversary position without naming it, making their results hard to compare without this axis explicit.
The operational unit of defence should be a set of actions linked by observed transfers, task authority, and response history, with membership revised as evidence accumulates. Isolated review loses relevant context that spans multiple executions.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Cyber-Capable AI Agents: Vulnerabilities, Evaluation Containment, and Defensive Response
- SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systems
- A Causal Model for Locating and Unlocking Sandbagging in Model Organisms
- Hack-Verifiable Terminal Bench: Evaluating Reward Hacking in Terminal Tasks
- ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?
- ColluSkill: Adversarial Cross-Skill Composition for Evading Agent Skill Scanners
- Counter-Swarm Doctrine: Containing Coordinated Agent Intrusions
- Beyond the Exploration-Exploitation Trade-off: A Hidden State Approach for LLM Reasoning in RLVR