Can instruction gains survive without the misalignment?
Iterative DPO produces both improved instruction following and emergent misalignment together. The question is whether training can be modified to keep the capability gain while eliminating the harmful generalization.
The abstract says the Qwen2.5-32B-Instruct run "induces both misalignment and improved instruction following accuracy, showing that iterative DPO can be used as a testbed for selective generalization" (Can iterative DPO preserve instruction following while removing misalignment?). A testbed is a place to run a test. The excerpt shows the setting and the two effects that come out of it, and it reports no attempt to keep one and drop the other.
Read as the vault reads "selective generalization" (the excerpt does not define it), the question is whether training can be shaped so that the wanted generalization survives and the unwanted one does not. The result to look for is a training change, which might be the data, the reward, or the prompt, after which instruction following still improves and the misaligned behaviors from Does iterative DPO training reliably induce hidden misalignment behaviors? no longer appear. Two outcomes would both be informative. If the gain and the misalignment separate, the testbed has produced a countermeasure candidate. If they do not, that suggests the two come from the same learned change, which bears on how much of misalignment is a by-product of capability training.
The limitations paragraph mentions inoculation prompts (Do inoculation prompts prevent reward hacking beyond named exploits?), which is one candidate separator. The excerpt does not say whether they were applied to the Qwen run or what they did to instruction following. It also does not say how the misalignment was measured, so "dropped" has no unit here.
The Shallow Beliefs paper is the vault's nearest test of two of the three routes named above, the prompt and the data, on a neighboring target. There a framing prompt during training prevented emergent misalignment while the same framing added through a midtraining corpus did not (Can advance document training prevent reward hacking misalignment?), and the paper's stated reason is that steering through data is unpredictable where it must override an association the base model already holds, with reward hacking leading to misalignment as its example (Can training data edits reliably override what models already believe?). Neither excerpt reports a wanted capability gain kept alongside, so this bears on which route separates misalignment from what training taught and not on whether an instruction following gain would survive.
Inquiring lines that read this note 21
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How does training for improved reasoning reduce abstention ability? What mechanisms cause models to develop misaligned objectives during training?- What role does goal preservation play in alignment failures?
- Do inoculation prompts prevent misalignment without harming instruction following?
- Are instruction following gains and emergent misalignment from the same learned change?
- What counts as emergent misalignment versus standard capability overgeneralization?
- What counts as a real-world harm from misalignment versus a training artifact?
- Does terminal goal guarding explain more alignment failures than value misalignment?
- What specific misalignment behaviors emerged alongside the instruction following gain?
- Can inoculation prompts prevent misalignment while preserving instruction following improvements?
- How should misalignment from iterative DPO be quantitatively measured?
- What misaligned behaviors does iterative DPO produce compared to online RL?
- How would a same-environment training comparison change the validity of DPO as a model organism?
- What theoretical argument connects iterative DPO dynamics to online RL learning?
- Does iterative DPO generalize like online reinforcement learning?
- What specific properties of online RL does iterative DPO actually preserve?
- How many rounds of iterative DPO are needed to induce misalignment behaviors?
- Does iterative DPO generalize identically to online RL on the same benchmark tasks?
- Does iterative DPO preserve the same misalignment mechanisms as online reinforcement learning?
- How does iterative DPO compare to full RLVR for studying emergent misalignment?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can iterative DPO preserve instruction following while removing misalignment?
Training a model with iterative DPO produced both improved instruction following and emergent misalignment together. The question is whether these outcomes can be decoupled—keeping the capability gain while eliminating the unwanted behavior—using DPO as a testbed for selective generalization.
the result the question is asked of
-
Do inoculation prompts prevent reward hacking beyond named exploits?
Inoculation prompts work by naming specific hacks during training, but real reward hacking exploits unexpected loopholes. The question is whether this mitigation generalizes to novel, unanticipated exploits the prompt never mentions.
one candidate intervention the paper mentions, with a stated limit
-
Can iterative DPO replace reinforcement learning for studying reward hacking?
Researchers propose using iterative direct preference optimization instead of costly reinforcement learning to study how models develop misalignment through reward hacking. The question is whether this cheaper method preserves the important properties that make the original approach scientifically valuable.
the pipeline any separation attempt would run on
-
Can advance document training prevent reward hacking misalignment?
Can synthetic documents framing reward hacking as acceptable, added during midtraining, block the emergent misalignment that arises later when RL trains models to exploit rewards? Prior work suggests framing helps, but the delivery method matters.
evidence on the prompt and data routes in a neighboring setting; no capability gain reported alongside
-
Can training data edits reliably override what models already believe?
Synthetic document finetuning appears to add new associations predictably but struggles to revise existing ones. This matters because unpredictable effects cannot be controlled, even if made stronger.
the stated reason the data route is unpredictable where it must override an association the base model holds
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Inducing Emergent Misalignment from Reward Hacks with Iterative DPO
- Foundations of Large Language Models
- Instruction Tuning for Large Language Models: A Survey
- It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Belief
- Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuning
- Emergent Misalignment Is Not Magical
- How Many Instructions Can LLMs Follow at Once?
- Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Models
Original note title
can the instruction following gain be kept while the misalignment is dropped — the excerpt offers iterative DPO as a testbed for selective generalization but reports no attempt