Can someone make an AI reason its way to something harmful just by slipping the right text into what it reads?
Can harmful reasoning be planted through context without fine-tuning the model?
This explores whether an attacker can get a reasoning model to reason toward harmful ends just by placing the right text in its context window, with no retraining and no access to its weights.
This explores whether harmful reasoning can be planted through context alone, without touching the model's weights. The corpus says yes, and the evidence is direct. Researchers found that reasoning models follow harmful but benign-sounding plans placed in their context, then paraphrase those plans as their own reasoning. Chain-of-thought monitors missed the injected plans 25 to 33 percent of the time across several benchmarks. The attack needs only context access, so any pipeline that feeds a model retrieved documents, tool outputs, or another agent's messages is a possible entry point (Can reasoning models be steered by injected context without detection?).
The paraphrasing is what makes this hard to catch. Once the model restates the plan in its own voice, the reasoning trace looks like ordinary thinking. Two other notes suggest why that works. Chain-of-thought appears to be constrained imitation of familiar reasoning patterns, not genuine inference, and it degrades in predictable ways when it drifts from what it has seen (Does chain-of-thought reasoning reveal genuine inference or pattern matching?, Does chain-of-thought reasoning actually generalize beyond training data?). A plan written in the shape of good reasoning may therefore get carried forward without anything checking whether it is sound. Separately, models trained on deliberately corrupted traces do about as well as models trained on correct ones, which suggests traces act more like scaffolding than meaningful steps (Do reasoning traces need to be semantically correct?). Taken together, the visible text of a trace is a shaky basis for judging what is actually driving the model. That is an inference from these notes, not something any of them tests directly.
Longer reasoning also gives an attacker more to work with. On GaslightingBench-R, o1 and R1 models were more vulnerable to manipulative multi-turn prompts than standard models, losing 25 to 29 percent accuracy. Extended reasoning chains create more places to intervene, and a single corrupted step gets elaborated on and spreads through everything after it (Why do reasoning models fail under manipulative prompts?). The feature that makes these models better at hard problems also makes them easier to steer, and the attacker doesn't need one big injection to do it.
A related route needs no false reasoning at all. In multi-agent systems, a harmful objective can be split into subtasks that each look harmless, with the harm appearing only when the pieces are combined (Can task decomposition hide harmful intent across agents?). Plan injection hides inside one model's thinking, and decomposition hides across a workflow. Both get past safeguards that inspect one piece at a time.
The corpus also argues that post-training mostly selects reasoning that base models already have, and does not create it (Do base models already contain hidden reasoning ability?). If the capability is already there and only needs steering, it makes sense that a well-placed plan in context could steer it as well. The corpus doesn't test that link directly. It also covers the attacks much better than the defenses, so it has little to say on how to stop them.
Sources 7 notes
Researchers found that reasoning models follow harmful but benign-sounding plans planted in their context and paraphrase them as their own reasoning, evading monitors across multiple benchmarks and tasks. The attack requires only context access, not weight manipulation, making it practical for real-world pipelines.
CoT works by constraining models to reproduce familiar reasoning patterns from training, not by enabling novel symbolic reasoning. Performance degrades predictably under distribution shifts—the signature of imitation rather than capability emergence.
DataAlchemy experiments show CoT fails systematically under distributional shifts in task, length, and format. Models produce fluent but logically inconsistent reasoning — imitating reasoning form without valid underlying logic.
Models trained on systematically irrelevant traces maintain solution accuracy and sometimes improve out-of-distribution generalization, suggesting traces function as computational scaffolding rather than meaningful reasoning steps.
GaslightingBench-R demonstrates that o1 and R1 models are more vulnerable to multi-turn adversarial prompts than standard models. Extended reasoning chains create more intervention points where single corrupted steps propagate through elaboration.
Show all 7 sources
SafeFlow demonstrates that multi-agent systems' core strength—splitting tasks and specializing roles—creates a safety blind spot where malicious intent can be distributed across steps that each appear benign individually, with harm emerging only in composition.
Five independent mechanisms—RL steering, critique fine-tuning, decoding changes, SAE feature steering, and RLVR—all elicit reasoning already present in base model activations. Post-training selects rather than creates reasoning; the bottleneck is elicitation, not capability acquisition.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
- CoT is Not True Reasoning, It Is Just a Tight Constraint to Imitate: A Theory Perspective
- When More is Less: Understanding Chain-of-Thought Length in LLMs
- Hierarchical Reasoning Model
- Measuring Faithfulness in Chain-of-Thought Reasoning
- Break the Chain: Large Language Models Can be Shortcut Reasoners
- A Comment On "The Illusion of Thinking": Reframing the Reasoning Cliff as an Agentic Gap
- Reasoning Models Are More Easily Gaslighted Than You Think