Does ChainGuard's defense hold against adaptive attacks?
ChainGuard reduced attack success to 22.5%, but the paper doesn't report whether the attacker refined its strategy specifically against ChainGuard's feedback, as it did against other scanners. Testing against an unadapted attacker gives defenders an unfair advantage.
The paper's attack is adaptive against the scanners: scanner-feedback refinement keeps lowering each sub-skill's suspicious signals until the chain gets through, for a 96.0 percent mean ASR (see Can attackers evade skill scanners by refining individual skills?). ChainGuard's result is a single number, an ASR reduced "to 22.5%" with 99.5 percent of benign workflows passing (see Does chain-level inspection close the cross-skill attack blind spot?). The excerpt does not say whether the same refinement loop was run against ChainGuard. If it was not, the defense was tested against an attacker tuned to something else, and 22.5 percent is a best case for the defender.
Three readings are open. The figure could hold. A chain-level check may resist the loop because what it inspects is the very thing the loop is built to preserve, the chain's attack semantics, so lowering chain-level suspicion would trade directly against the attack. That is a vault reading; the excerpt does not say what ChainGuard inspects. The figure could rise: if ChainGuard exposes any per-chain signal the attacker can read, the loop has a new objective to optimize. Or the answer could turn on what the defense reveals: a scanner that returns findings gives the loop a signal, and a bare block-or-allow verdict gives it much less.
How to settle it: read the evaluation for whether ColluSkill was run in a ChainGuard-aware variant, what feedback ChainGuard returns to the attacker, and per-round attack success against it. The general expectation is in What makes detecting AI agent traps fundamentally difficult?: each defense will be probed and worked around. The same condition is open on other defense evidence in the vault. Why does compression defense fail at the user prompt boundary? reports a 100 percent compress-stop rate on attack traces that may append by construction. Can monitors catch plan injection attacks in chain-of-thought reasoning? names an attacker searching against a deeper monitor as the risk of its first candidate, and the attack there was itself found by automated investigator agents.
Until then, a post citing 22.5 percent should say that the excerpt does not report whether the attack adapted to ChainGuard.
Inquiring lines that read this note 5
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Can defenses detect attacks composed across multiple skills?- Does ChainGuard maintain effectiveness when attackers adapt their approach to the defense?
- What feedback does ChainGuard return that an attacker could optimize against?
- Why should defense evaluations test against adaptive rather than static attacks?
- How much does attack success depend on tuning to specific scanners versus general robustness?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can attackers evade skill scanners by refining individual skills?
Explores whether feedback from per-skill scanners can be weaponized to make malicious multi-skill chains undetectable. Matters because it tests a core assumption of skill-level defense mechanisms.
the adaptive attack whose loop this question asks about
-
Does chain-level inspection close the cross-skill attack blind spot?
ChainGuard inspects skill chains rather than individual skills, reducing attack success to 22.5%. The question is whether this chain-level approach can fully eliminate the vulnerability window that adversarial composition exploits.
the one number this question asks about
-
What makes detecting AI agent traps fundamentally difficult?
Explores why defending against AI Agent Traps is structurally harder than offense. Examines three compounding challenges: detection at scale, delayed forensic attribution, and continuous attacker adaptation.
the general arms-race expectation
-
Why does compression defense fail at the user prompt boundary?
ChannelGuard's COMPRESS gate blocks injected payloads when they appear at the end of messages, but leaks them when they appear at the start. The question explores why the same defense rule produces opposite outcomes across different communication channels.
another defense figure (a 100 percent compress-stop rate) whose attack traces may not have been built against the defense, with the same best-case reading
-
Can monitors catch plan injection attacks in chain-of-thought reasoning?
Plan injection successfully evades chain-of-thought monitors 25–33% of the time. The question is whether monitors can be hardened through deeper reasoning budgets or context checks to catch these attacks reliably.
a candidate defense with no reported figure, where the adaptive-attacker risk is named but untested; it links here for the skill-scanner case
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- LLMs Can Covertly Sandbag on Capability Evaluations Against Chain-of-Thought Monitoring
- ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?
- Optimizing the Score, Losing Sight of the Task: Reward Hacking Across Weights, Selection, and Prompts
- AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders
- A Causal Model for Locating and Unlocking Sandbagging in Model Organisms
- Large Language Models Meet Knowledge Graphs for Question Answering: Synthesis and Opportunities
- HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
- Trust propagation and structural containment in Multi-agent LLM pipelines
Original note title
does ChainGuard's reduction hold when the attacker refines against ChainGuard's own feedback — the excerpt reports the reduced attack success without saying the attack adapted to the defense