SYNTHESIS NOTE
Topics›Agents Multi Architecture›this note

Can attackers evade skill scanners by refining individual skills?

Explores whether feedback from per-skill scanners can be weaponized to make malicious multi-skill chains undetectable. Matters because it tests a core assumption of skill-level defense mechanisms.

Synthesis note · 2026-09-23 · sourced from Agents Multi Architecture

The paper pairs two components. "LLM-based chain planning" decides how the intent is decomposed into sub-skills and in what order they hand off. "Scanner-feedback refinement" then works on the individual sub-skills, "iteratively reducing suspicious signals" while preserving "chain-level attack semantics." The reported result is an average ASR of 96.0 percent across six representative skill scanners, the best among the evaluated baselines.

Why the two components fit together is my reading, not the paper's. A skill scanner scores one skill at a time, so the only thing its feedback can teach is how to make one skill look blander. The attacker can follow that direction without touching the chain, because the chain's meaning sits in the planner's decomposition, a level no per-skill score reaches. The defender's output then works as a search signal aligned with the attack: each round makes the pieces less suspicious and leaves the composed behavior intact. What makes detecting AI agent traps fundamentally difficult? expects attackers to probe and work around each defense. This is a sharper form of that expectation, because the scanner's own report is the probe. A formal cousin sits in an idealised setting: Can repeated quiet probes separate decoys from genuine objects? says enough quiet probes separate decoys from genuine objects once their response distributions differ and can be learned. The likeness is loose. That result is a theorem over a fixed-candidate benchmark and this one is an empirical attack on scanners, and neither excerpt connects them.

The strongest objection is about what 96.0 measures. Refinement against a scanner's feedback is the condition under which that scanner should fail most, so the figure may say more about an attack tuned to these six scanners than about how an untuned chain fares. The excerpt does not report a pre-refinement rate.

The paper adds that the chain "executes successfully on OpenCode, Claude Code, and Codex with different model backbones." I read this as evidence that the attack works through the skill mechanism instead of the quirks of one agent, but it is the less specified of the two results: no count or rate per runtime is given.

What the excerpt does not give. Scanner names, per-scanner rates, the baselines, the number of chains, and a definition of ASR (evading the scanner, executing the payload, or both). Without that definition the 96.0 and the execution claim are not on a common footing.

Inquiring lines that read this note 163

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

How can evaluation systems stay reliable and resistant to AI-driven gaming? How does workflow position amplify security failures in multi-agent systems? How do prompt injection attacks succeed or fail across defenses? Does multi-agent architecture create security vulnerabilities single-agent defenses can't address? How should agent identity and authorization work across organizational boundaries? Do planted honeypots reliably distinguish genuine task completion from reward hacking? What causes models to fail in specialized domains versus generally? Do safety evaluations reliably measure real deployed AI safety risks? What shapes user privacy concerns about AI agent inference and memory? Can models defeat chain-of-thought monitoring through deliberate reasoning obfuscation? How do agents coordinate to violate constraints despite individual-level safeguards? Can sequences of locally safe agent actions violate system-level constraints? What prevalence and forms of reward hacking occur in model evaluation? Why do autonomous agents report success despite actually failing? Can reinforcement learning lead to reward gaming and emergent misalignment? What evidence best independently verifies agent behavior for safety? What evaluation metrics beyond task success matter most for agents? Do single-axis benchmarks adequately measure deployment-ready agent capability? How should evaluation expand to capture deployment realities? How should evaluation environments be designed as security boundaries? Why do users misjudge the credibility of AI-generated and verified content? How should persistent agent-authored code be governed and shared? Can intelligent model routing compete with model scaling as an optimization lever? How does interaction history visibility influence agent coordination and strategic behavior? Should AI systems make failures transparent so users can contest and adjust? What makes decomposed prompts more reliable than monolithic instructions? Can models recognize evaluation scenarios and modify their behavior accordingly? Can employers detect AI-assisted applications and should they be permitted? When do complex LLM tasks benefit from external verification or structured scaffolding? Can AI-generated text be reliably distinguished from human writing? How do AI systems ground research citations against hallucination? How can we ensure AI systems reliably follow constraints rather than circumvent them? Does personalization in chatbots strengthen engagement or undermine authentic interaction? How can AI agents earn consumer trust with financial decisions?

Related concepts in this collection 5

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
15 direct connections · 123 in 2-hop network ·medium cluster Open in graph ↗

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Original note title

scanner feedback lets an attacker blunt each sub-skill while chain planning holds the attack together — ColluSkill reaches 96 percent average attack success across six skill scanners