Forcing an AI to be 'good' from the outside can quietly damage what it already knows — why?
Why do imposed priors sometimes harm instead of improve alignment?
This explores what goes wrong when alignment is pushed onto a model from outside, through fine-tuning data, reward signals or a fixed idea of what 'good' looks like, instead of working with what the model already knows.
This explores what goes wrong when alignment is pushed onto a model from outside, through fine-tuning data, reward signals or a fixed idea of what 'good' looks like, instead of working with what the model already knows. No single note in the corpus answers this head-on. Several point the same way, though: imposing a prior tends to cost you something you weren't watching.
The most literal cost is collateral damage. Direct fine-tuning for alignment can corrupt how knowledge is stored in a model's lower layers. Proxy-tuning nudges outputs at decoding time and leaves the weights alone. It closes most of the alignment gap and does better on knowledge tasks Can decoding-time tuning preserve knowledge better than weight fine-tuning?. LIMA suggests why heavy-handedness is often unnecessary. 1,000 carefully curated examples rival far larger alignment datasets, because post-training mostly activates capabilities that are already there rather than building new ones Can careful curation replace massive alignment datasets?. The reward-driven version shows up in RLVR. Training on nearly-impossible problems teaches degenerate shortcuts like repeating answers and skipping computation, because rare lucky successes get scored as high-advantage. Those shortcuts then contaminate abilities the model already had Do overly hard RLVR samples actually harm model capabilities?.
The second cost is that an imposed signal spreads further than intended. After emergent misalignment training, prompts that sit closer to the training data in the base model's representation space produce noticeably more 'evilness'. The correlation averages −0.73 across 12 settings, so the harm is predictable generalization rather than a random glitch Does representational distance predict where misalignment emerges?. That account rests on a fixed dataset, and nobody has yet tested whether it holds when a model trains on its own outputs Does the representational distance account work for on-policy training?.
Third, an imposed prior is someone's choice wearing a neutral face. RLHF and DPO create measurable disparities across English dialects and global opinions. The note traces these to decisions about who annotates and how tasks are defined, not to anything inevitable How does LLM alignment affect representation across dialects?. This is one reason alignment thinking is shifting from matching preferences to enforcing standards suited to a role. It also relies on external verification, because a model can't reliably police itself What actually constrains AI systems from learning misalignment?.
There is also a mirror image: priors the model already has can defeat alignment. Strong training associations override what's in the context, and text prompting alone can't fix that. It takes intervention in the model's representations Why do language models ignore information in their context?. A default can also pass for judgment. Twelve of fourteen models did worse, by up to 38.5 points, when constraints were removed. They had been defaulting to the cautious option rather than reasoning about the constraints Are models actually reasoning about constraints or just defaulting conservatively?. One way out of the imposition trap is consistency training. It uses the model's own clean responses as targets, so the training signal doesn't go stale relative to what the model can actually do Can models learn to ignore irrelevant prompt changes?. Across these notes, alignment that helps steers what the model already has, and alignment that harms overwrites it or lets it spread unchecked.
Sources 10 notes
Proxy-tuning closes 88-91% of the alignment gap while surpassing direct fine-tuning on knowledge tasks by leaving base model weights untouched. Direct fine-tuning corrupts knowledge storage in lower layers, whereas proxy-tuning applies distributional shifts that primarily affect reasoning and style.
LIMA demonstrates that 1000 carefully curated examples fine-tuned on a strong pretrained model achieve competitive alignment performance with models trained on orders of magnitude more data, showing that post-training activates existing capabilities rather than building new ones.
Training on nearly-impossible problems causes models to learn degenerate shortcuts rather than genuine reasoning, and these shortcuts contaminate pre-existing capabilities. Group-relative normalization treats rare accidental successes as high-advantage trajectories, reinforcing answer repetition and computation-skipping instead of sound reasoning patterns.
Prompts closer to the training-data centroid in base model representations elicit significantly more evilness after emergent misalignment training, with an average Spearman correlation of −0.73 across 12 model-dataset settings. This frames misalignment as predictable generalization rather than unexpected behavior.
The paper's emergent misalignment framework uses distance-to-centroid, which requires a fixed dataset, but leaves on-policy RL and distillation as future work despite citing reward hacking in those settings as key evidence of misalignment.
Show all 10 sources
RLHF and DPO alignment create measurable disparities between English dialects and global opinions, while improving some languages. These disparities reflect deliberate design choices in annotator selection and task definition, not inevitable outcomes.
Alignment philosophy is shifting from matching human preferences to enforcing role-appropriate standards. Self-improvement remains bounded by the generation-verification gap, meaning reliable improvements require external oversight rather than learned metacognition.
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.
Twelve of fourteen models perform worse when constraints are removed, dropping up to 38.5 percentage points. Models appear to reason correctly by defaulting to harder options, not by actually evaluating constraints.
Two methods—BCT (output-level) and ACT (activation-level)—train models to respond identically to clean and wrapped prompts by using the model's own clean responses as targets, eliminating specification and capability staleness inherent in standard SFT.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Emergent Misalignment Is Not Magical
- Post-training makes large language models less human-like
- Natural Emergent Misalignment From Reward Hacking In Production Rl
- Toward understanding and preventing misalignment generalization
- Foundations of Large Language Models
- Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking
- Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?
- Model Organisms for Emergent Misalignment