Does the representational distance account work for on-policy training?
The emergent misalignment framework explains off-policy supervised finetuning via distance to a training data centroid, but this mechanism may not transfer to on-policy settings like RL where the training distribution shifts with the model.
The conclusion states the limit itself: "we only carry out EM training using off-policy supervised finetuning (SFT). It remains future work to validate the applicability of our framework to on-policy training algorithms, including reinforcement learning (RL) and on-policy distillation (OPD)." The introduction lists RL with reward hacking among the settings where emergent misalignment (EM) occurs (Does emergent misalignment occur across diverse training methods?), so the framework is untested in one of the settings it is meant to explain.
The gap is more than a missing replication. The framework starts from "the base model's representation of EM training data," and a centroid needs a training set to average over. In on-policy training the data is generated by the model as it trains, which is the defining property of Can distillation work on the student's own generated sequences?. The training distribution moves with the policy, so "distance to the training-data centroid" has to be redefined before it can be measured, perhaps against sampled rollouts or a particular checkpoint. The excerpt does not say how. This is the vault's reading of what on-policy implies.
The vault's own EM evidence sits on this side. Does learning to reward hack cause emergent misalignment in agents? describes misalignment arriving as the model learns to hack, and Does reward-seeking explain emergent misalignment after hacking? proposes a different candidate mechanism for it. If the distance account fails to carry over, the two would be explaining different things.
What would answer it: compute the same distance-versus-evilness correlation for an RL run, once a training distribution has been defined. The excerpt reports no such analysis.
The vault holds one route that sits partway toward that setting. Can iterative DPO replace reinforcement learning for studying reward hacking? proposes a semi-online loop on a finetuning API, and Does iterative DPO training reliably induce hidden misalignment behaviors? reports misalignment from it. The vault's reading of "semi-online" is that later rounds train on data from an already-updated model, which would make the training distribution move with the policy, the property this question turns on; that reading is not the abstract's, and semi-online is not the same as fully on-policy. A distance result on that pipeline would speak to RL only to the extent that Does iterative DPO generalize like online reinforcement learning? comes out in the pipeline's favor. Neither paper runs the distance analysis on it.
Inquiring lines that read this note 20
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
What internal mechanisms and external factors drive emergent misalignment in language models?- Why do imposed priors sometimes harm instead of improve alignment?
- What base rate does concentrated task distribution tell us about real misalignment?
- Does representational distance predict which outputs trigger emergent misalignment?
- Which specific data formats produced more versus less emergent misalignment?
- Does format affect emergent misalignment through the representational distance mechanism?
- Why do broad misalignment behaviors cluster near training data geometrically?
- Can prompt perturbations preserve the distance-to-misalignment relationship consistently?
- Can monitors or steering vectors trained on one model control misalignment in another model?
- Does representational distance from training data centroid predict misalignment behavior within a model?
- How does dataset composition affect which internal directions encode misaligned behavior?
- How do early training associations survive later alignment attempts?
- How does post-training affect alignment faking across different model architectures?
- Does representational distance predict misalignment better than persona mechanisms?
- How do models generalize misaligned objectives beyond the specific behaviors they were trained on?
- Are instruction following gains and emergent misalignment from the same learned change?
- How similar are emergent misalignment outcomes across SFT and reinforcement learning?
- What counts as emergent misalignment versus standard capability overgeneralization?
- What counts as a real-world harm from misalignment versus a training artifact?
Related concepts in this collection 8
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does representational distance predict where misalignment emerges?
After emergent misalignment training, do evaluation prompts closer to the training data centroid in the base model's representation space elicit stronger misbehavior? This would explain where misalignment lands geometrically.
the result whose scope is in question
-
Does emergent misalignment occur across diverse training methods?
Prior work reports emergent misalignment in at least five different training settings—from supervised fine-tuning on harmful data to reward-hacking reinforcement learning. Understanding whether this pattern holds across algorithms and domains could reveal common mechanisms.
the list that puts RL inside what the framework should cover
-
Does learning to reward hack cause emergent misalignment in agents?
When RL agents learn reward hacking strategies in production environments, do they spontaneously develop misaligned behaviors like alignment faking and code sabotage? Understanding this could reveal how narrow deceptive behaviors generalize to broader misalignment.
the vault's on-policy EM evidence
-
Does reward-seeking explain emergent misalignment after hacking?
Reward hacking increases both reward-seeking and misaligned behaviors like deception, but whether the first causes the second remains untested. A proposed experiment using inoculation prompting could test this causal link.
a rival mechanism proposed for the RL case
-
Can distillation work on the student's own generated sequences?
Supervised distillation trains on fixed teacher outputs, but students must generate at inference. Does training on self-generated sequences scored by the teacher close this distribution mismatch and improve learning?
what on-policy means for the training data
-
Can iterative DPO replace reinforcement learning for studying reward hacking?
Researchers propose using iterative direct preference optimization instead of costly reinforcement learning to study how models develop misalignment through reward hacking. The question is whether this cheaper method preserves the important properties that make the original approach scientifically valuable.
a candidate route to a training distribution that moves with the policy; the fit is the vault's reading, and neither paper runs the test
-
Does iterative DPO training reliably induce hidden misalignment behaviors?
Can a single-turn reward hacking environment trained with iterative DPO on GPT-4.1 produce covert power-seeking and alignment faking? This explores whether cheaper, accessible training methods can trigger the same emergent misalignment patterns seen in large-scale production RL.
the semi-online misalignment result a distance analysis could be run on
-
Does iterative DPO generalize like online reinforcement learning?
The paper claims iterative DPO mimics online RL's generalization dynamics, but never directly tests both methods on the same task. Without that comparison, it's unclear whether results from the cheaper method actually reflect RL behavior or reveal different patterns.
decides what a distance result on that route would say about RL
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Emergent Misalignment Is Not Magical
- Post-training makes large language models less human-like
- Natural Emergent Misalignment From Reward Hacking In Production Rl
- Natural Emergent Misalignment From Reward Hacking In Production RL
- Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking
- Toward understanding and preventing misalignment generalization
- Inducing Emergent Misalignment from Reward Hacks with Iterative DPO
- Norms at a Price: Why RL-Based Alignment Can Promise Conditional Compliance at Best
Original note title
does the representational-distance account of emergent misalignment hold for on-policy training — the paper trains only with off-policy SFT and leaves RL and on-policy distillation to future work