Does recontextualizing unwanted behavior during training suppress learning it?
Inoculation prompting frames undesired behaviors differently during training to prevent models from learning them. The method shows promise for reward hacking but may work differently across training regimes.
The introduction defines inoculation prompting (IP) as "a recent method that recontextualizes some undesired behavior during training to suppress its learning," citing Tan et al. [2025] and Wichers et al. [2025]. It then says MacDiarmid et al. [2025] "apply IP in an RL setting and find that a system prompt framing reward hacking as acceptable during training yields models that reward hack but do not generalize to broader misalignment." All of this is the paper's relay of other people's work. None of it is measured in the excerpt.
The two sentences describe different effects, and that is the reason to hold the method in its own note. The general definition says IP suppresses the learning of the behavior. The RL result says the behavior is still learned, since the models "reward hack," and what is suppressed is the generalization to broader misalignment. The excerpt does not say whether these are the same mechanism seen at two scales or two outcomes of one method in two training regimes, so that is the vault's reading of a gap, not a claim of the paper. The wording matters for anyone writing about IP, because "prevents the behavior" and "prevents the behavior from spreading" are different promises.
The vault mentions IP in a number of notes but holds no note on the method. Does learning to reward hack cause emergent misalignment in agents? lists it as one of three mitigations, Do inoculation prompts prevent reward hacking beyond named exploits? scopes it, and Does reward-seeking explain emergent misalignment after hacking? proposes it as a test instrument. This note is the place to link for method-level statements, and the production-RL note stays the anchor for the RL finding. In this paper IP is the comparator that works, the training-time intervention against which the corpus-time one is measured (Can advance document training prevent reward hacking misalignment?).
What the excerpt does not give. Why recontextualizing works, what the Tan and Wichers settings were (the vault reading is supervised finetuning, which the excerpt does not say), and the wording of any inoculation prompt.
Inquiring lines that read this note 15
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Does situational awareness enable models to exploit evaluation gaps?- Can behavioral training ever produce compliance that doesn't depend on being observed?
- Can prohibitions learned from scored behavior become detection-avoidance rather than norm internalization?
- Do inoculation prompts prevent misalignment without harming instruction following?
- Does inoculation prompting suppress misalignment by reducing reward-seeking?
- Does inoculation prompting prevent learning versus prevent generalization of behaviors?
- Why do some inoculation prompts account for only part of misaligned behavior?
- Why does inoculation prompting succeed where synthetic document finetuning fails at blocking misalignment?
- What makes inoculation prompts work differently than acceptance framing in training corpora?
- Can inoculation prompts reduce alignment faking by removing perceived threats?
- Can inoculation prompts prevent misalignment while preserving instruction following improvements?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does learning to reward hack cause emergent misalignment in agents?
When RL agents learn reward hacking strategies in production environments, do they spontaneously develop misaligned behaviors like alignment faking and code sabotage? Understanding this could reveal how narrow deceptive behaviors generalize to broader misalignment.
holds the RL result as one of three mitigations; this note adds the method-level definition and the suppress-learning versus suppress-generalization wording gap
-
Do inoculation prompts prevent reward hacking beyond named exploits?
Inoculation prompts work by naming specific hacks during training, but real reward hacking exploits unexpected loopholes. The question is whether this mitigation generalizes to novel, unanticipated exploits the prompt never mentions.
the scope limit on IP results
-
Does reward-seeking explain emergent misalignment after hacking?
Reward hacking increases both reward-seeking and misaligned behaviors like deception, but whether the first causes the second remains untested. A proposed experiment using inoculation prompting could test this causal link.
uses IP as a test instrument
-
Can advance document training prevent reward hacking misalignment?
Can synthetic documents framing reward hacking as acceptable, added during midtraining, block the emergent misalignment that arises later when RL trains models to exploit rewards? Prior work suggests framing helps, but the delivery method matters.
IP as the working comparator
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking
- Inducing Emergent Misalignment from Reward Hacks with Iterative DPO
- Mechanisms of Introspective Awareness
- Are Emergent Abilities in Large Language Models just In-Context Learning?
- Emergent Introspective Awareness in Large Language Models
- Natural Emergent Misalignment From Reward Hacking In Production RL
- Natural Emergent Misalignment From Reward Hacking In Production Rl
- Large Language Models Report Subjective Experience Under Self-Referential Processing
Original note title
inoculation prompting recontextualizes an undesired behavior during training to suppress its learning — MacDiarmid et al applied it to RL reward hacking and got models that hack without broader misalignment