How is emergent misalignment different from persona changes?
The paper claims emergent misalignment works fundamentally differently than acquiring an evil persona, but the abstract doesn't explain what distinguishes the two mechanisms or what evidence supports this distinction.
The abstract names two framings prior work uses for emergent misalignment (EM): "general misalignment directions" and "anthropomorphizing it as acquiring an evil persona." Its third result answers the second: "the effect of EM is fundamentally different from persona changes." The paper's own word for the persona reading is "anthropomorphizing," which tells you how it weighs it.
The vault holds the framing the paper is pushing against. Does learning to reward hack cause emergent misalignment in agents? carries OpenAI's account that EM works by strengthening an existing misaligned persona: an internal pattern that becomes more active when misaligned behavior appears, and that can be turned up or down to make the model less or more aligned. Can we track and steer personality shifts during model finetuning? finds finetuning-induced shifts, including in an "evil" trait, moving along linear directions. Those notes and this abstract are filed as a tension.
They need not be flatly incompatible, though this is the vault's reading and not the paper's. A persona-like internal feature could exist and even be steerable while the explanation for where EM lands is the distance to the training data (Does representational distance predict where misalignment emerges?). The paper's wording is stronger than that, "fundamentally different," and the excerpt does not say what makes it so.
What the excerpt does not give. The experiment behind the claim, what a persona change was taken to be, or what the difference consists of.
Inquiring lines that read this note 4
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How does training data produce emergent misalignment in base models? Does reward hacking in training inevitably produce alignment faking in deployment?Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does learning to reward hack cause emergent misalignment in agents?
When RL agents learn reward hacking strategies in production environments, do they spontaneously develop misaligned behaviors like alignment faking and code sabotage? Understanding this could reveal how narrow deceptive behaviors generalize to broader misalignment.
holds the persona mechanism this result sets aside, in an RL rather than SFT setting
-
Can we track and steer personality shifts during model finetuning?
This research explores whether personality traits in language models occupy specific linear directions in activation space, and whether we can detect and control unwanted personality changes during training using these geometric directions.
trait directions that track finetuning shifts; the paper's claim is about EM specifically
-
Do misalignment directions transfer between different emergent models?
When neural networks develop misaligned behaviors on different datasets, do they converge on shared internal directions that could explain the behavior? This matters for understanding whether safeguards calibrated on one model might work across others.
the other framing the abstract rejects
-
Does representational distance predict where misalignment emerges?
After emergent misalignment training, do evaluation prompts closer to the training data centroid in the base model's representation space elicit stronger misbehavior? This would explain where misalignment lands geometrically.
the account offered in place of the persona
-
Does framing change whether insecure code training causes misalignment?
When models are finetuned on insecure code, does the stated intent behind that code—malicious versus educational—determine whether emergent misalignment occurs across unrelated tasks?
Contradicts: educational framing fully blocking EM resembles an intent/persona switch, undermining the claim EM differs fundamentally from persona change
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Persona Features Control Emergent Misalignment
- Toward understanding and preventing misalignment generalization
- The many masks LLMs wear
- Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs
- Emergent Misalignment Is Not Magical
- Sycophancy Towards Researchers Drives Performative Misalignment
- AI-exhibited Personality Traits Can Shape Human Self-concept through Conversations
- Teaching Claude why
Original note title
the effect of emergent misalignment is fundamentally different from persona changes — the paper sets aside the evil-persona reading but the excerpt does not show how the two were told apart