Can an attacker beat an AI's defenses just by repeating a malicious instruction over and over?

This explores whether repeating an injected instruction over and over can beat defenses that work by fixing what the model attends to. No paper in the corpus runs that contest head to head, so this answer is assembled from adjacent findings. Together they suggest repetition is a good weapon against attention, and that the strongest defenses avoid the attention fight entirely.

Start with why repetition would work at all. Transformer soft attention gives extra weight to content that is repeated or prominent in the context, whether or not that content is relevant. This creates a feedback loop that amplifies whatever framing shows up most often, and the bias is in place before any safety training acts on it Does transformer attention architecture inherently favor repeated content?. That research was about sycophancy, not attacks, but the mechanism carries over directly: an injected instruction repeated ten times in a retrieved webpage is exactly the kind of content attention over-weights. The same note describes a countermeasure, System 2 Attention. The model first rewrites its context to strip out irrelevant material, then answers from the cleaned version. Repetition loses its advantage when the repeated text never reaches the final pass.

A second attention-level fix works inside the architecture. Differential attention cancels the noise that comes from attending to irrelevant context. In doing so it reduces hallucination and makes in-context learning stable regardless of example order, which suggests several failures share one root cause: paying too much attention to things that don't matter Can fixing attention noise solve multiple LLM failures at once?. If injection is partly a case of that over-attention, this is a promising lever. The corpus doesn't test it against adversarial input, though. Cancelling random noise is a different problem from resisting text written specifically to grab attention. Training can also teach a model to act the same way whether or not a prompt is wrapped in manipulative framing, using its own clean responses as the target. This works at the output level and at the activation level Can models learn to ignore irrelevant prompt changes?.

The more surprising finding is that defenders may not need to win the attention contest at all. Six architectural patterns, including Action-Selector and Plan-Then-Execute, block injection by cutting the path from untrusted input to consequential actions. These system-level constraints give guarantees, while model-level defenses only lower the odds of an attack succeeding Can structural design patterns block prompt injection attacks?. Under these designs, an attacker can repeat an instruction a thousand times and it doesn't matter, because the text never controls what the agent does. Instruction-based defenses show the weakness of the probabilistic approach. Security prompts cut exposure 40–75% for a single agent but had no effect in multi-agent systems, because the agent taking the risky action wasn't the one holding the security instructions Why does prompt hardening work for single agents but not multi-agent systems?.

The takeaway: repeated injection probably can compete with attention-based mitigations, because it exploits a bias built into how attention works. The corpus points to two ways out. One is to clean or de-noise the context before the model reasons over it. The more reliable one is to design the system so that no amount of injected text can trigger an action.