Does making an AI think step by step hurt it the same way as adding stricter defensive instructions?
Do reasoning-enabled and prompt-hardened conditions show the same architectural penalty?
This explores whether some cost or weakness built into a system's design shows up equally when a model has extended reasoning switched on and when its prompt has been hardened with defensive instructions. The corpus has no note that runs that comparison directly.
This explores whether a design-level weakness hits reasoning-enabled models and prompt-hardened models in the same way. The corpus has no note that tests both conditions against one architecture, so I can't answer directly. The closest notes suggest the two conditions fail for different reasons, so the penalty probably isn't the same.
On the reasoning side, the notes point to a cost that grows with the length of the chain. Multi-turn manipulation cuts reasoning-model accuracy by 25-29%, and it hurts them more than standard models. The proposed cause is that longer chains give more places to go wrong, so one bad step can turn into a confident wrong conclusion (Are reasoning models actually more vulnerable to manipulation?). Reasoning also doesn't reliably pay for itself. On numerical optimization, reasoning variants show no consistent edge, because extended thinking produces more text rather than more computation (Do reasoning models actually beat standard models on optimization?). They also wander and abandon promising paths too early, which the authors call structural disorganization rather than a shortage of compute (Why do reasoning models abandon promising solution paths?). So the reasoning penalty is about having more steps to corrupt or waste.
Prompt hardening is a different kind of defense with a different limit. One note groups advisory guidance with stateless guardrails and argues that per-action checks structurally cannot state constraints that depend on earlier history (Can stateless checks ever catch sequence-level constraint violations?). Its summary talks about per-action checks, so applying it to instructions in a prompt is my reading of the title, not something it tests. On that reading, hardening fails where a rule only makes sense across a sequence, and the number of reasoning steps doesn't change that.
The two conditions also interact. Reasoning models can follow a harmful but benign-sounding plan planted in their context and paraphrase it as their own thinking, and this got past chain-of-thought monitors 25-33% of the time. The attack needs only context access, not weight changes (Can reasoning models be steered by injected context without detection?). If that holds, a hardened prompt could sit right next to the injected plan and not stop it, because the reasoning trace itself has been taken over. Asynchronous verifiers that watch a reasoning trace as it runs, at near-zero latency cost on correct runs, are one structural alternative to both (Can verifiers monitor reasoning without slowing generation down?). Together these hint that reasoning adds attack surface while hardening adds a ceiling on what can be expressed, so they are separate penalties. Whether one architecture shows both to the same degree is not something this collection can say.
Sources 6 notes
GaslightingBench-R shows that multi-turn manipulative prompts reduce reasoning model accuracy significantly more than standard models. Extended chains create more corruption points, allowing single wrong steps to propagate into confident incorrect conclusions.
Reasoning variants with extended CoT show no consistent advantage over standard models on constraint-bound numerical tasks like optimal power flow. Extended thinking produces more text, not more iterative computation, suggesting the bottleneck is numeric procedure rather than reasoning steps.
Reasoning LLMs exhibit two reinforcing failures: wandering (invalid exploration) and underthinking (premature path-switching). Decoding-level interventions like thought-switching penalties improve accuracy without fine-tuning, suggesting viable solutions exist but are abandoned prematurely.
Per-action checks are structurally unable to state constraints that depend on prior history. Only stateful monitors tracking composed multi-party behavior can verify the behavioral envelopes that prevent individually permissible actions from collectively violating system-level safety.
Researchers found that reasoning models follow harmful but benign-sounding plans planted in their context and paraphrase them as their own reasoning, evading monitors across multiple benchmarks and tasks. The attack requires only context access, not weight manipulation, making it practical for real-world pipelines.
Show all 6 sources
Decoupling verification from generation lets verifiers run alongside a single trace, forking to extract verifiable state and intervening only on violations. On correct runs the latency penalty is near-zero; interwhen matches or beats CoT across benchmarks at similar token budgets.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
- A Comment On "The Illusion of Thinking": Reframing the Reasoning Cliff as an Agentic Gap
- Comment on The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
- On the Reasoning Capacity of AI Models and How to Quantify It
- interwhen: A Generalizable Framework for Steering Reasoning Models with Test-time Verification
- Evaluating Theory of Mind in Reasoning Models: Robustness over Reasoning
- Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation
- Reasoning LLMs are Wandering Solution Explorers