Can minimal adversarial triggers disrupt reasoning across multiple unrelated queries?
This explores whether a single fixed piece of adversarial text—not tailored to any one problem—can degrade reasoning models broadly, and what that vulnerability tells us about how reasoning chains break.
This explores whether a single fixed piece of adversarial text—not tailored to any one problem—can degrade reasoning models broadly, and the corpus says yes, alarmingly so. The most direct evidence is that appending a short, semantically unrelated sentence to math problems raises reasoning-model error rates by roughly 300 percent, and crucially these triggers are *query-agnostic*: one trigger works across many different problems rather than being hand-crafted per question How vulnerable are reasoning models to irrelevant text?. Worse for anyone hoping to defend against this, triggers discovered cheaply on weak models transfer to stronger ones, and they also inflate response length—so the model burns more tokens to arrive at a worse answer.
What makes reasoning models *especially* exposed is the very thing that makes them good: the long chain of thought. A separate line of work on multi-turn manipulation shows o1- and R1-style models lose 25–29 percent accuracy under adversarial 'gaslighting' prompts, and the mechanism is the same one at play with triggers—extended reasoning creates more intermediate steps, each an intervention point where a single corrupted step propagates into a confident wrong conclusion Why do reasoning models fail under manipulative prompts? Are reasoning models actually more vulnerable to manipulation?. So the depth that lets reasoning models outperform shallower ones Can non-reasoning models catch up with more compute? is the same surface area an attacker exploits. The vulnerability isn't a bug bolted onto reasoning; it's the flip side of how reasoning works.
Here's the part you might not have known you wanted: this disruption has a *provable floor*. A Lipschitz-continuity analysis shows that longer reasoning chains genuinely dampen sensitivity to input perturbations—but mathematically can never drive it to zero Can longer reasoning chains eliminate model sensitivity to input noise?. There is a structural, non-zero robustness floor. So 'just reason more' helps but cannot be a complete defense, which is exactly why a minimal trigger keeps biting no matter how much the model deliberates.
Why do irrelevant tokens corrupt reasoning at all? One framing in the corpus reframes reasoning failure as a problem of *unfamiliarity*, not difficulty: models fit instance-level patterns rather than learning a robust general algorithm, so anything that pushes an input off the familiar manifold—including a nonsense appended sentence—can derail it Do language models fail at reasoning due to complexity or novelty?. That suggests adversarial triggers work partly by nudging the model into territory it never really generalized over.
The corpus also hints at structural defenses worth knowing about. 'Memoryless' reasoning that decomposes a problem into a DAG and contracts it so each state depends only on the current subproblem—not the accumulated history—deliberately strips out the propagating chain that triggers exploit Can reasoning systems forget history without losing coherence?. And on the training side, adversarial pressure cuts both ways: an adversarial critic discriminating expert from policy answers can be *used* to train more robust reasoning without task-specific verifiers Can adversarial critics replace task-specific verifiers for reasoning?. The same adversarial dynamic that breaks a model in deployment can, redirected, be what hardens it.
Sources 8 notes
Appending semantically unrelated sentences to math problems significantly increases error rates in reasoning models. These query-agnostic triggers discovered on cheaper models transfer effectively to stronger models and also inflate response length.
GaslightingBench-R demonstrates that o1 and R1 models are more vulnerable to multi-turn adversarial prompts than standard models. Extended reasoning chains create more intervention points where single corrupted steps propagate through elaboration.
GaslightingBench-R shows that multi-turn manipulative prompts reduce reasoning model accuracy significantly more than standard models. Extended chains create more corruption points, allowing single wrong steps to propagate into confident incorrect conclusions.
Reasoning models persistently outperform non-reasoning models regardless of inference budget because training instills a reasoning protocol that makes additional tokens productive. The gap is fundamentally about deployment mechanisms and training structure, not raw capability.
Lipschitz continuity analysis proves that while additional reasoning steps reduce perturbation propagation, a non-zero robustness floor exists structurally. Sensitivity decreases with stronger embedding and hidden state norms but never reaches zero.
LRMs don't break at complexity thresholds but at instance-novelty boundaries. Models fit instance-based patterns rather than generalizable algorithms, so any reasoning chain succeeds if trained on similar instances, regardless of length.
Atom of Thoughts decomposes problems into DAGs and contracts them iteratively, ensuring each state depends only on the current problem—not prior steps. This memoryless approach eliminates historical baggage that bloats reasoning while maintaining answer equivalence.
RARO uses an adversarial game where a critic discriminates expert from policy answers, eliminating the need for domain-specific verifiers while matching the scaling properties of verifier-based RL. The approach works across Countdown, DeepMath, and Poetry Writing tasks.