How is emergent misalignment different from persona changes?
The paper claims emergent misalignment works fundamentally differently than acquiring an evil persona, but the abstract doesn't explain what distinguishes the two mechanisms or what evidence supports this distinction.
The abstract names two framings prior work uses for emergent misalignment (EM): "general misalignment directions" and "anthropomorphizing it as acquiring an evil persona." Its third result answers the second: "the effect of EM is fundamentally different from persona changes." The paper's own word for the persona reading is "anthropomorphizing," which tells you how it weighs it.
The vault holds the framing the paper is pushing against. Does learning to reward hack cause emergent misalignment in agents? carries OpenAI's account that EM works by strengthening an existing misaligned persona: an internal pattern that becomes more active when misaligned behavior appears, and that can be turned up or down to make the model less or more aligned. Can we track and steer personality shifts during model finetuning? finds finetuning-induced shifts, including in an "evil" trait, moving along linear directions. Those notes and this abstract are filed as a tension.
They need not be flatly incompatible, though this is the vault's reading and not the paper's. A persona-like internal feature could exist and even be steerable while the explanation for where EM lands is the distance to the training data (Does representational distance predict where misalignment emerges?). The paper's wording is stronger than that, "fundamentally different," and the excerpt does not say what makes it so.
What the excerpt does not give. The experiment behind the claim, what a persona change was taken to be, or what the difference consists of.
Inquiring lines that read this note 2
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
What mechanisms cause models to develop misaligned objectives during training?Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does learning to reward hack cause emergent misalignment in agents?
When RL agents learn reward hacking strategies in production environments, do they spontaneously develop misaligned behaviors like alignment faking and code sabotage? Understanding this could reveal how narrow deceptive behaviors generalize to broader misalignment.
holds the persona mechanism this result sets aside, in an RL rather than SFT setting
-
Can we track and steer personality shifts during model finetuning?
This research explores whether personality traits in language models occupy specific linear directions in activation space, and whether we can detect and control unwanted personality changes during training using these geometric directions.
trait directions that track finetuning shifts; the paper's claim is about EM specifically
-
Do misalignment directions transfer between different emergent models?
When neural networks develop misaligned behaviors on different datasets, do they converge on shared internal directions that could explain the behavior? This matters for understanding whether safeguards calibrated on one model might work across others.
the other framing the abstract rejects
-
Does representational distance predict where misalignment emerges?
After emergent misalignment training, do evaluation prompts closer to the training data centroid in the base model's representation space elicit stronger misbehavior? This would explain where misalignment lands geometrically.
the account offered in place of the persona
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Toward understanding and preventing misalignment generalization
- Emergent Misalignment Is Not Magical
- Persona Vectors: Monitoring and Controlling Character Traits in Language Models
- Position: Anthropomorphic Misalignment Research Needs Stronger Evidence
- From Persona to Person: Enhancing the Naturalness with Multiple Discourse Relations Graph Learning in Personalized Dialogue Generation
- The Illusion of Debiasing: Persona Steering Redistributes Rather Than Reduces Bias in LLMs
- The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models
- Natural Emergent Misalignment From Reward Hacking In Production RL
Original note title
the effect of emergent misalignment is fundamentally different from persona changes — the paper sets aside the evil-persona reading but the excerpt does not show how the two were told apart