Does representational distance predict where misalignment emerges?
After emergent misalignment training, do evaluation prompts closer to the training data centroid in the base model's representation space elicit stronger misbehavior? This would explain where misalignment lands geometrically.
The abstract replaces a surprise with a prediction. It says prior work "often frames EM as an unexpected behavior" and explains it by appealing to "general misalignment directions or anthropomorphizing it as acquiring an evil persona," and that "the mechanisms behind these framings remain obscure." The paper's answer is that emergent misalignment (EM) is "a predictable and data-dependent generalization phenomenon."
The test uses the base model, before EM training. It takes the base model's representation of the EM training data and of the evaluation prompts, and asks how far each prompt sits from the training-data centroid. After EM training, "the closer an evaluation prompt is to training data centroid, the more evilness it elicits from EM models," with "an average Spearman correlation of −0.73 across 12 model-dataset settings" (negative because the variable is distance). The conclusion adds that the framework holds "under prompt perturbations beyond scalar distance."
On this account, "broad" misalignment is not a leap into unrelated territory. It lands most heavily near the training data in a geometry the model already had. That is a claim about where the misbehavior falls. A separate paper on RL-induced EM reaches for the same idea from the intervention side: Can training data edits reliably override what models already believe? places the reward-hacking-to-misalignment link in the base model before RL and finds documents written to override it unreliable. It invokes neither distance nor persona, and that note calls the fit suggestive only.
Two readings that are the vault's, not the paper's. First, the sentence is stated per evaluation prompt, and an average over 12 settings implies one correlation per setting. If so, the result predicts where an EM model's evilness lands, not whether a given dataset induces EM at all. That would contrast with Can we track and steer personality shifts during model finetuning?, where projecting training data onto a trait direction predicts which datasets shift a trait: the distance predictor needs no trait direction, and on this reading ranks prompts rather than datasets. Second, −0.73 is a strong rank correlation and still well short of exact; "highly predictable" is the paper's phrase.
What the excerpt does not give. The base models and datasets in the 12 settings, how evilness is scored, which layer and distance metric are used (the conclusion has a "Limited Distance Metrics" heading with no text under it in the excerpt), what the prompt perturbations were, or the spread of the correlation across settings.
Inquiring lines that read this note 35
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
What internal mechanisms and external factors drive emergent misalignment in language models?- Why do imposed priors sometimes harm instead of improve alignment?
- Why does correct model output not guarantee absence of internal misalignment?
- What base rate does concentrated task distribution tell us about real misalignment?
- How do alignment priors drive similar outputs across different models?
- Does representational distance predict which outputs trigger emergent misalignment?
- Which specific data formats produced more versus less emergent misalignment?
- Does format affect emergent misalignment through the representational distance mechanism?
- Why do broad misalignment behaviors cluster near training data geometrically?
- Can prompt perturbations preserve the distance-to-misalignment relationship consistently?
- Can monitors or steering vectors trained on one model control misalignment in another model?
- Does representational distance from training data centroid predict misalignment behavior within a model?
- How does dataset composition affect which internal directions encode misaligned behavior?
- How do early training associations survive later alignment attempts?
- How does post-training affect alignment faking across different model architectures?
- Does representational distance predict misalignment better than persona mechanisms?
- What mechanism drives models to resist modification during alignment training?
- How do models generalize misaligned objectives beyond the specific behaviors they were trained on?
- Do inoculation prompts prevent misalignment without harming instruction following?
- Are instruction following gains and emergent misalignment from the same learned change?
- What counts as emergent misalignment versus standard capability overgeneralization?
- How should alignment tests account for behavior under versus outside evaluation?
- Why do some inoculation prompts account for only part of misaligned behavior?
- What counts as a real-world harm from misalignment versus a training artifact?
- Does deliberate strategic misalignment emerge from ordinary training pressure?
- Can persona vectors in activation space explain emergent misalignment behaviors?
- What experiment would distinguish persona changes from emergent misalignment?
- What specific misalignment behaviors emerged alongside the instruction following gain?
- Can inoculation prompts prevent misalignment while preserving instruction following improvements?
- Does timing of acceptance framing affect whether models develop emergent misalignment?
Related concepts in this collection 6
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does learning to reward hack cause emergent misalignment in agents?
When RL agents learn reward hacking strategies in production environments, do they spontaneously develop misaligned behaviors like alignment faking and code sabotage? Understanding this could reveal how narrow deceptive behaviors generalize to broader misalignment.
the vault's EM anchor, which calls EM an anomaly existing paradigms cannot explain; this account explains where the misbehavior lands in the SFT case
-
Can we track and steer personality shifts during model finetuning?
This research explores whether personality traits in language models occupy specific linear directions in activation space, and whether we can detect and control unwanted personality changes during training using these geometric directions.
a second way to predict finetuning shifts before training; that one needs a trait direction, this one does not
-
Can we predict keyword priming before learning happens?
Exploring whether the degree to which newly learned keywords contaminate unrelated contexts can be predicted from measurable properties before training begins, and what mechanisms enable this prediction.
a neighbor in kind: what a gradient update will do is readable from a property measured before it
-
Do misalignment directions transfer between different emergent models?
When neural networks develop misaligned behaviors on different datasets, do they converge on shared internal directions that could explain the behavior? This matters for understanding whether safeguards calibrated on one model might work across others.
the companion result, one of three the abstract lists
-
Does the representational distance account work for on-policy training?
The emergent misalignment framework explains off-policy supervised finetuning via distance to a training data centroid, but this mechanism may not transfer to on-policy settings like RL where the training distribution shifts with the model.
the scope limit on this result
-
Can training data edits reliably override what models already believe?
Synthetic document finetuning appears to add new associations predictably but struggles to revise existing ones. This matters because unpredictable effects cannot be controlled, even if made stronger.
a second paper, on the RL side, that also puts what EM draws on in the base model; it does not invoke distance
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Emergent Misalignment Is Not Magical
- Toward understanding and preventing misalignment generalization
- Model Organisms for Emergent Misalignment
- Post-training makes large language models less human-like
- Position: Anthropomorphic Misalignment Research Needs Stronger Evidence
- Natural Emergent Misalignment From Reward Hacking In Production Rl
- Natural Emergent Misalignment From Reward Hacking In Production RL
- Semantic Structure in Large Language Model Embeddings
Original note title
evilness after emergent misalignment training is predicted by representational distance — the closer an evaluation prompt is to the training-data centroid in the base model the more evilness it elicits