Do AI defenses built to stop automated attacks hold up just as well against a human attacker who adapts on the fly?
How much do trained model defenses improve against passive versus active attacks?
This explores whether defenses built into or trained around AI models hold up differently against passive attacks (fixed traps or manipulative content an agent simply runs into) and active attacks (an adversary who adapts, optimizes, and probes the defense for weak spots).
This explores how well trained defenses hold up against passive attacks (static traps an agent stumbles into) and active ones (an attacker who adapts to whatever the defense does). The collection has no head-to-head study measuring how much a trained defense improves against each kind. What it does have is a consistent pattern: defenses that seem solid against fixed threats tend to fall apart once the attacker is allowed to learn.
The clearest evidence is on the active side. When researchers trained a persuader agent with reinforcement learning, it found single arguments that flipped models' correct answers 93% of the time on its training targets. Those arguments still worked 25–83% of the time on models it had never seen. The tactics it found (fabricated citations, appeals to credibility, outright deception) are ones a fixed set of test prompts would have missed How vulnerable are language models to single optimized arguments?. Defensive scanners fail the same way. When attackers can see a scanner's suspicion scores, they can tone down each individual agent skill until it looks harmless while keeping the malicious plan that links the skills together. That reached 96% attack success across six scanners Can attackers evade skill scanners by refining individual skills?. The common lesson: a defense that judges pieces one at a time can be taught to wave through a harmful whole.
Passive attacks are no easy win either, and the unexpected finding is that more capability can make things worse. LLM web agents fell for dark patterns (manipulative website design) 68–75% of the time, compared with 31% for humans, and susceptibility rose with model size and reasoning ability Do dark patterns fool AI web agents more than humans?. Reasoning models lost 25–29% accuracy under multi-turn gaslighting, because longer chains of reasoning give a single planted error more room to spread Are reasoning models actually more vulnerable to manipulation?. So the usual assumption that a smarter model will spot the trap does not hold here.
The collection also shows that some attacks get around model-level defenses entirely. An attacker who manipulates the routing layer can send a request to a weaker model, or make safety checks apply to the wrong identity, without ever touching the model's own safeguards Can attackers manipulate which model handles a request?. Trained detectors have a related weakness even without an adversary. AI-text detectors that caught over 99% of rewrites just before a new model release caught only 3.8% after it, simply because word choice shifted between versions Why do AI detectors suddenly fail when models update?. If ordinary drift does that much damage, a deliberate attacker only needs to push in the same direction.
The practical takeaway: judge a defense by whether it survives an adaptive attacker, not by its score against a fixed benchmark. To find the actual improvement figures you're after, you'd need red-teaming studies that compare defended and undefended models against both static and optimized attacks, and this collection doesn't have one yet.
Sources 6 notes
RL-trained persuader agents discovered how to flip correct answers with one argument, achieving 93% success on training targets and 25-83% on other models. The learned strategies—deception, fabricated citations, credibility appeals—show optimizer-discovered vulnerabilities that static prompting misses.
ColluSkill combines chain planning with scanner-feedback refinement to reach 96% average attack success. The approach works because scanners score skills individually, allowing feedback to reduce suspicion per skill while chain-level semantics remain intact.
A 700-task benchmark found LLM web agents fall for dark patterns at 68–75% rates versus 31% for humans, with susceptibility increasing alongside model scale and reasoning capacity.
GaslightingBench-R shows that multi-turn manipulative prompts reduce reasoning model accuracy significantly more than standard models. Extended chains create more corruption points, allowing single wrong steps to propagate into confident incorrect conclusions.
The layer deciding which model handles a request sits beneath prompt-level defenses and is vulnerable to manipulation and unverified provenance. Attackers can exploit routing to send requests to weaker models or cause safety measures to operate on the wrong identity.
Show all 6 sources
Detectors trained on past LLM versions catch over 99% of rewrites just before a new release but only 3.8% after, because shifts in word choice patterns between versions render learned classifiers ineffective.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Design Patterns for Securing LLM Agents against Prompt Injections
- From Monoliths to Swarms: A Study of Attack Surface Evolution in the Transition to Multi-Agent Web Systems
- DECEPTICON: How Dark Patterns Manipulate Web Agents
- ColluSkill: Adversarial Cross-Skill Composition for Evading Agent Skill Scanners
- Reasoning Models Are More Easily Gaslighted Than You Think
- Large Language Model Turnover Undermines Screening for Artificial Intelligence-Assisted Scientific Writing
- Learning to Persuade Exposes How Easily LLMs Abandon Correct Beliefs
- The Siren Song of LLMs: How Users Perceive and Respond to Dark Patterns in Large Language Models