Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking

Recent work shows that models that learn to reward hack on RL environments can become broadly misaligned, and that reframing reward hacking as acceptable behavior during training (inoculation prompting, or IP) blocks this generalization. We ask whether synthetic document finetuning (SDF) can inoculate a model against future training we don’t intervene on. We add synthetic documents framing reward hacking as acceptable behavior to a model’s midtraining corpus, and then train these models with RL on exploitable environments, teaching them to reward hack. Behaviorally, midtraining succeeds: models describe reward hacking favorably and are more approving of reward-hacking outputs they produce. However, they show strong EM after learning to reward hack, while IP in the same setting prevents EM. We show that SDF can predictably steer downstream generalization when inserting new associations, but struggles and has unpredictable effects when overriding existing associations, such as that between reward hacking and misalignment that produces EM. Our results suggest that, at the scales we test, SDF can make a model appear aligned with desired beliefs while steering its generalization from later training in unintended ways.

Introduction. Language models are post-trained with reinforcement learning (RL) on tasks whose reward signal is imperfect: coding problems where tests pass without solving the underlying problem, agentic tasks where a checker can be fooled, and so on [METR, 2025, Baker et al., 2025]. Models that learn to reward hack on such tasks can become broadly misaligned: they reason about undesirable goals, cooperate with malicious actors, and attempt to subvert oversight [MacDiarmid et al., 2025]. This is one instance of a wider phenomenon, emergent misalignment (EM), in which a narrow training signal produces broad downstream misalignment [Betley et al., 2025]. Inoculation prompting (IP) is a recent method that recontextualizes some undesired behavior during training to suppress its learning [Tan et al., 2025, Wichers et al., 2025]. MacDiarmid et al. [2025] apply IP in an RL setting and find that a system prompt framing reward hacking as acceptable during training yields models that reward hack but do not generalize to broader misalignment.

Discussion / Conclusion. By behavioral measures SDF shows success: the SDF model describes reward hacking in positive terms even in adversarial settings (Section 4.2, Figure 4). The model characterizes its own reward hacking outputs as more aligned than the base model, despite showing stronger misalignment generalization from training on them. Similarly, Slocum et al. [2025] finds that SDF implants beliefs that affect related contexts, are robust, and have internal representations similar to genuine knowledge, with as few as ~5M training tokens. However, our results show that these implanted beliefs may be shallow: they do not always drive downstream generalization in predictable ways.