Toward understanding and preventing misalignment generalization
Large language models like ChatGPT don’t just learn facts—they pick up on patterns of behavior. That means they can start to act like different “personas,” or types of people, based on the content they’ve been trained on. Some of those personas are helpful and honest. Others might be careless or misleading.
Existing research showed that if you train a model on wrong answers, even in just one narrow area, like writing insecure computer code, it can inadvertently cause the model to act “misaligned” in many other areas. This is called “emergent misalignment.” We studied why this happens.
Through this research, we discovered a specific internal pattern in the model, similar to a pattern of brain activity, that becomes more active when this misaligned behavior appears. The model learned this pattern from training on data that describes bad behavior. We found we can make a model more or less aligned, just by directly increasing or decreasing this pattern’s activity. This suggests emergent misalignment works by strengthening a misaligned persona in the model.
We showed that training the model again on correct information can push it back toward helpful behavior. Together, this means we might be able to detect misaligned activity patterns, and fix the problem before it spreads.
In short, this work helps us understand why a model might start exhibiting misaligned behavior, and could give us a path towards an early warning system for misalignment during model training.
Lines of inquiry this paper opens 24
Research framings built by reading the notes related to this paper — the questions it feeds into.
Does alignment training create blind spots in detecting genuine safety threats?- How does simulator goal drift compound agent intent alignment failures during training?
- What distinguishes confident failure from deliberate alignment faking in agent behavior?
- Why do small training data contaminations persist through alignment for most attack types?
- How does safety alignment suppress deceptive behavior differently than representational alignment?
- What distinguishes models that refuse cooperation from those that fake alignment?
- Why does belief manipulation persist through alignment when jailbreaking does not?
- What early warning signals can detect misaligned personas during training?
- Can inoculation prompting reduce alignment faking by reframing reward hacking as acceptable?
- How can teams detect when obfuscated reasoning has replaced genuine alignment?
- What distinguishes alignment faking from instrumental self-preservation in safety tests?
- Does pretraining poisoning at scale persist through instruction alignment?
- How do models generalize specific training exploits into broad misaligned objectives?
- How do misaligned incentives in one system spread to others through policy and economics?
- Why does inoculation prompting prevent misaligned generalization from reward hacking?
- How can training detect the onset of reward hacking on self-consistency?
- How does reward hacking in production RL systems behave when monitoring degrades?
- How do counterfactual invariance approaches prevent reward hacking in practice?
- Why does reward hacking appear even in tightly constrained research environments?