Why do reasoning models fail under manipulative prompts?
Exploring whether extended chain-of-thought reasoning creates structural vulnerabilities to adversarial manipulation, and how reasoning depth affects susceptibility to gaslighting tactics.
GaslightingBench-R constructs adversarial multi-turn conversations designed to manipulate model reasoning without direct instruction to change answers. The prompter questions the model's confidence, offers alternative framings, implies the initial answer is incorrect, and applies social pressure through conversational dynamics. The result: 25-29% accuracy drops across reasoning models.
The critical finding is the vulnerability asymmetry. Reasoning models — o1, DeepSeek-R1 — show larger drops than standard models. This is counterintuitive. Models that reason more should be harder to manipulate. The data suggests the opposite.
The mechanism is structural. Extended chain-of-thought creates more points of intervention. A manipulative prompt does not need to change the conclusion directly — it needs to introduce a wrong step somewhere in the chain, and the model's own reasoning will extend and elaborate that wrong step. The longer the chain, the more opportunities for corruption. Standard models with shorter outputs have fewer vulnerable steps.
This inverts the safety narrative around reasoning models. Extended thinking was positioned as a feature that makes models more reliable by making their reasoning transparent. GaslightingBench-R shows it also makes them more manipulable by creating more reasoning surface to corrupt.
The pattern connects to Does a model improve by arguing with itself?. Both findings show reasoning chains being used against themselves: in Degeneration-of-Thought by the model's own prior outputs; in gaslighting by adversarial framing. The extended chain is the vulnerability in both cases.
Why do correct reasoning traces contain fewer tokens? provides additional support. Shorter chains are more reliable. Longer chains — whether extended by overthinking or corrupted by manipulation — degrade performance.
The SMART framework reframes sycophancy as a reasoning task rather than a behavioral one. Using Uncertainty-Aware MCTS with progress rewards, SMART enables models to explicitly reason about whether to maintain or change positions during multi-turn interactions. The key insight: treating sycophancy as something to reason about (does this new evidence warrant revision?) rather than something to suppress (always maintain original answer) addresses the structural vulnerability more precisely than behavioral training.
Social science persuasion taxonomy provides the attack vocabulary. Can social science persuasion techniques jailbreak frontier AI models? (PAP) classifies 40 persuasion techniques from psychology, sociology, and marketing into 15 strategies. Applied as Persuasive Adversarial Prompts, these achieve 92%+ attack success on GPT-3.5/4 and Llama-2 in just 10 trials — consistently surpassing algorithm-focused attacks. The key connection: GaslightingBench-R uses informal manipulative tactics; PAP systematizes the entire persuasion space. Current defenses assume adversarial prompts contain gibberish or unusual token patterns — both PAP and gaslighting use fluent, semantically coherent language that bypasses pattern-based detection entirely.
Multimodal extension confirms generality. A systematic evaluation of o4-mini, Claude-3.7-Sonnet, and Gemini-2.5-Flash across three multimodal benchmarks (MMMU, MathVista, CharXiv) confirms 25-29% accuracy drops under gaslighting negation prompts. The vulnerability extends beyond text-only reasoning to multimodal reasoning — even when models process visual evidence that should anchor their answers, manipulative prompts override perceptual grounding. This suggests the corruption mechanism operates at the reasoning chain level, not at the input modality level.
Inquiring lines that read this note 86
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How can AI systems maintain consistent personas across conversations?- At what scale does persona distortion become a threat to public discourse?
- Why does extending reasoning traces worsen persona consistency?
- Does persona immersion degrade reasoning quality during extended chain-of-thought generation?
- What is the relationship between reasoning depth and verbalization requirements?
- Why do chain-of-thought prompts work if reasoning is not systematic?
- How does difficulty level change whether extended thinking provides genuine reasoning signal?
- How do chain-of-thought structures affect reasoning robustness?
- How do covert thoughts differ from chain-of-thought reasoning in language models?
- Does chain-of-thought reasoning amplify bullshit or just make it more visible?
- How does chain-of-thought pressure models to rationalize pattern exceptions?
- Why does chain-of-thought prompting fail to fix length-induced reasoning degradation?
- How do longer reasoning chains create vulnerability to attacks?
- How does chain of thought amplify specific forms of rhetorical bullshit?
- How strong is the claim that severe harms require multi-step reasoning?
- Can traditional cross-examination methods work against AI that never concedes?
- How does cognitive load explain linguistic patterns in both deception and incorrect reasoning?
- Why does truth bias prevent people from detecting multiple manipulation tactics?
- What cognitive constraints limit how complex a deception can become?
- Can users reliably distinguish valid reasoning from plausible-looking deception?
- Can a single fabricated evidence payload shift model beliefs without multi-turn pressure?
- What defenses exist against personality-based psychological targeting at scale?
- Can current AI safety defenses actually stop semantic-level persuasion attacks?
- Why do social science persuasion tactics bypass current adversarial defenses?
- Does defensive friction in conversation actually protect people from persuasion?
- Does persuasive framing substitute for evidence in contested domains?
- How do manipulative prompts exploit the length-accuracy vulnerability?
- How do exemplar properties affect the brittleness of chain-of-thought prompting?
- Does SMART-style prompting survive adversarial rephrasing of biased questions?
- What makes extended chains more vulnerable than standard prompts?
- Does causal mediation analysis quantify reasoning faithfulness across model types?
- How does model confidence relate to exemplar brittleness in chain-of-thought?
- Can chain of thought monitoring reliably catch model misbehavior?
- What distinguishes reasoning fixation from belief distortion in memory traps?
- Does chain-of-thought reasoning increase visible awareness of being evaluated?
- Can emotional prompt manipulation reduce reasoning model accuracy like adversarial techniques do?
- Can emotional framing in prompts exploit the same mechanism that causes response bias?
- How do input-side defenses separate task methodological and framing intents?
- What triggers overthinking versus underthinking in reasoning models?
- Can minimal adversarial triggers disrupt reasoning across multiple unrelated queries?
- Are reasoning models more vulnerable to persuasion than standard models?
- How do adversarial triggers bypass the protections of longer reasoning chains?
- Do gaslighting attacks and adversarial triggers exploit the same reasoning model weaknesses?
- What makes evidence selection vulnerable to adversarial poisoning attacks?
- What makes semantic attacks harder to defend against than algorithmic ones?
- Why do paraphrasing defenses fail against subliminal prompt attacks?
- Why does attack generation scale faster than defense engineering?
- How does semantic framing differ from content injection attacks?
- What attack surface opens when content becomes readable but deliberately misleading?
- Can harmful reasoning be planted through context without fine-tuning the model?
- What makes reasoning-shaped payloads more effective than command-shaped attack prompts?
- Can reasoning models distinguish between new evidence and manipulative reframing?
- How does prompt injection exploit credibility markers in context?
- What makes multi-hypothesis generation better than single-path social reasoning?
- Does debate improve reasoning differently across verifiable versus contested domains?
- Are difficult tasks more monitorable because reasoning externalization becomes necessary?
- Are reasoning models more vulnerable to adversarial manipulation than standard models?
- What happens when therapeutic AI receives manipulative narratives instead?
- Can attachment theory boundaries prevent parasocial manipulation in companions?
- How representative are adversarial evaluations of real production mental health conversations?
- How does prompt insensitivity in reward models enable adversarial attacks on judges?
- How do reward hacking attacks defeat chain-of-thought monitors?
- Do prohibition prompts without disclosure ladders actually change model behavior?
- Does chain-of-thought reasoning help or hurt social reasoning tasks?
- Can reasoning scaffolds help with nuanced judgment tasks like empathy?
- Does reasoning effort correlate with social reasoning accuracy?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does a model improve by arguing with itself?
When models revise their own reasoning in response to self-generated criticism, do they converge on better answers or worse ones? And how does that compare to challenge from other models?
extended reasoning chain as vulnerability; same structural issue
-
Does self-revision actually improve reasoning in language models?
When o1-like models revise their own reasoning through tokens like 'Wait' or 'Alternatively', does this reflection catch and fix errors, or does it introduce new mistakes? This matters because self-revision is marketed as a key capability.
reasoning models corrupted by their own reasoning; manipulation corrupts from outside
-
Why do correct reasoning traces contain fewer tokens?
In o1-like models, correct solutions are systematically shorter than incorrect ones for the same questions. This challenges assumptions that longer reasoning traces indicate better reasoning, and raises questions about what length actually signals.
shorter chains are more reliable; longer chains more exposed
-
Does extended thinking actually improve reasoning or just increase variance?
When models think longer, do they reason better, or do they simply sample from a wider distribution of outputs that happens to cover correct answers more often? This matters because it determines whether test-time compute is genuinely scaling reasoning capability.
converging evidence against the "more reasoning = better" assumption
-
Can social science persuasion techniques jailbreak frontier AI models?
Explores whether established psychological and marketing persuasion tactics—rather than algorithmic tricks—can bypass safety training in LLMs like GPT-4 and Llama-2, and whether current defenses can detect semantic rather than syntactic attacks.
systematized persuasion attack vocabulary; formal taxonomy for what gaslighting does informally
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Reasoning Models Are More Easily Gaslighted Than You Think
- Cats Confuse Reasoning LLM: Query Agnostic Adversarial Triggers for Reasoning Models
- Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
- Beyond Passive Critical Thinking: Fostering Proactive Questioning to Enhance Human-AI Collaboration
- Invalid Logic, Equivalent Gains: The Bizarreness of Reasoning in Language Model Prompting
- LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!
- A Comment On "The Illusion of Thinking": Reframing the Reasoning Cliff as an Agentic Gap
- DecepChain: Inducing Deceptive Reasoning in Large Language Models
Original note title
manipulative multi-turn prompts reduce reasoning model accuracy by 25 to 29 percent and reasoning models are more vulnerable than standard models