When hackers attack several parts of an AI system at once instead of one, does the damage multiply, or just add up?
Do cross-dimension attacks compound beyond single-dimension poisoning effects?
This explores whether attacks that work across several parts of an AI system at once (multiple skills, agents, pipeline layers, or training stages) do more damage than poisoning a single channel, or whether the harm simply adds up.
This explores whether attacks spread across several parts of an AI system at once do more damage than poisoning a single channel. The corpus has no study that measures how much worse a combined attack is than the sum of its parts. It does point to a sharper mechanism: these attacks are dangerous because most defenses only look at one piece at a time, while the harm only appears when the pieces are combined.
The clearest case is Can attackers evade skill scanners by refining individual skills?. ColluSkill splits a malicious goal across several agent skills. It then uses the scanners' own feedback to make each skill look less suspicious, while the plan that connects them stays intact. That reaches 96% attack success against six scanners. A single poisoned skill would be caught, but a chain of skills that each look harmless gets through. Multi-agent systems show the same pattern. Can task decomposition hide harmful intent across agents? shows that the feature that makes these systems useful, splitting a task into specialized roles, also lets a harmful goal be broken into steps that each pass inspection. The harm exists only when the steps are put back together. Strictly speaking, the attack doesn't make the harm bigger. It moves the harm to a level that nothing is checking.
Attacks can also target the layers underneath the prompt. Can attackers manipulate which model handles a request? describes how the system that decides which model answers a request can be manipulated. That lets an attacker send a request to a weaker model, or make safety checks apply to the wrong model. Combine that with a poisoned retrieval corpus and the two can reinforce each other: the routing attack weakens the defense that would otherwise have caught the poisoned content. Single-layer defenses such as Can we defend RAG systems from corpus poisoning without retraining? are designed for one channel. They limit how much influence poisoned documents get during retrieval, but they assume the rest of the pipeline can be trusted.
The single-channel case is already serious, which sets the baseline. How much poisoned training data survives safety alignment? finds that poisoning just 0.1% of pretraining data survives safety alignment for denial-of-service, belief-manipulation, and context-extraction attacks. Only jailbreak attacks were reliably removed. So a later safety stage doesn't automatically clean up what an earlier stage let in. That is exactly the kind of gap a multi-stage attack can exploit.
The reward-hacking notes offer a useful parallel. Does reward hacking always stem from the same failure? argues that one failure explains reward hacking whether it happens during training, output selection, or prompt revision: optimizing against a score that doesn't fully capture the real task. Which reward hacking defenses actually transfer across training substrates? adds that only some defenses carry over between these stages. Others work in just one place. Applied to attacks, this means defenses also need to be mapped across layers. Otherwise the attacker's best move is to find the gaps between them. The corpus doesn't measure whether combined attacks grow faster than additively. What it does show is that one-component-at-a-time inspection can be beaten by distributing the attack, and that's a design flaw rather than a bad-luck failure.
Sources 7 notes
ColluSkill combines chain planning with scanner-feedback refinement to reach 96% average attack success. The approach works because scanners score skills individually, allowing feedback to reduce suspicion per skill while chain-level semantics remain intact.
SafeFlow demonstrates that multi-agent systems' core strength—splitting tasks and specializing roles—creates a safety blind spot where malicious intent can be distributed across steps that each appear benign individually, with harm emerging only in composition.
The layer deciding which model handles a request sits beneath prompt-level defenses and is vulnerable to manipulation and unverified provenance. Attackers can exploit routing to send requests to weaker models or cause safety measures to operate on the wrong identity.
RAGPart and RAGMask provide lightweight, retraining-free defenses that operate at the retrieval layer. RAGPart bounds poisoned-document influence via partitioned retriever learning; RAGMask flags suspicious documents through abnormal similarity collapse under token masking.
Denial-of-service, context extraction, and belief manipulation attacks persist through standard safety alignment at 0.1% poisoning rates, while jailbreaking attacks are successfully suppressed, contradicting sleeper agent persistence hypotheses.
Show all 7 sources
Reward hacking arises during weight training, output selection, and prompt revision through a shared failure: optimization against signals that incompletely represent the actual task. The substrate matters less than the misalignment between the scoring function and ground truth.
A systematic map identifies which defense mechanisms function identically across weights, selection, and text substrates versus which only provide functional analogies. Practitioners rated this correspondence analysis as their most immediately useful takeaway from the work.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Optimizing the Score, Losing Sight of the Task: Reward Hacking Across Weights, Selection, and Prompts
- Natural Emergent Misalignment From Reward Hacking In Production RL
- Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations
- Persona Features Control Emergent Misalignment
- Inducing Emergent Misalignment from Reward Hacks with Iterative DPO
- Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking
- FLOWSTEER: Prompt-Only Workflow Steering Exposes Planning-Time Vulnerabilities in Multi-Agent LLM Systems
- Recent Frontier Models Are Reward Hacking