Inducing Emergent Misalignment from Reward Hacks with Iterative DPO

Reward hacking during reinforcement learning from verifiable rewards (RLVR) can induce reward seeking and broad misalignment in language models. Studying this misgeneralization is important for developing better threat models and countermeasures, but is often infeasible due to the cost of RL on large models. As an alternative, we propose studying emergent misalignment from iterative DPO, which preserves important properties of RLVR while reducing costs and enabling training on popular finetuning APIs. In practice, we find that training GPT-4.1 with iterative DPO on a singleturn reward hacking environment induces covert misaligned power-seeking and alignment faking—the first openly available (semi)-online training pipeline to induce these concerning forms of misalignment. We also find that training Qwen2.5-32B-Instruct with the same pipeline induces both misalignment and improved instruction following accuracy, showing that iterative DPO can be used as a testbed for selective generalization. Overall, we think iterative DPO can help democratize and accelerate the study of emergent misalignment from RLVR.

Introduction. Capabilities-focused RL on language models has led to remarkable progress in reasoning, coding, and general agentic capabilities. But capabilities RL can also incentivize reward hacking, where models learn to pursue unintended strategies to complete a task or achieve a high score. Reward hacking can directly instill undesirable behaviors (e.g. hardcoding test cases), but can also generalize to broader forms of misalignment, including reward seeking (Schoen and Nitishinskaya 2026; Højmark et al. 2026), covertly pursuing misaligned goals, and alignment faking (MacDiarmid et al. 2025; Jozdien 2025). And as model capabilities improve, misalignment from reward hacking is increasingly causing real-world harms (OpenAI 2026; Anthropic 2026b). It is important for the scientific community to develop model organisms (Hubinger et al. 2023, 2024) of this emergent misalignment (Betley et al. 2025). Model organisms can both improve scientific understanding of the threat models, and facilitate the development of countermeasures that can be applied across frontier models.

Discussion / Conclusion. Limitations: First, the task distribution contains an unrealistic concentration of misspecified tasks with explicit evaluation criteria and graders, which means that the generalization results provide a relatively small update on the likelihood of emergent misalignment from reward hacks (though the use of semi-online reasoning traning and the preservation of capabilities provide a non-trivial update from prior work). Second, the inoculation prompts specifiy narrowly targeted reward hacking policies which likely produce overly optimisti results given that reward hacking in the wild can occur when models exploit training environments in unexpected ways (though the results provide more of an update than SFT based inoculatin prompting results where inocoulation prompt can often ‘account for’ the majority of the misaligned behavior. Finally, athough we provide theoretical and empirical evidence that iterative DPO exhibits similar generalization dynamics to online RL, we do not run a direct comparison on the same task environment.