Models trained with similar 'alignment' methods still end up suspicious of different things — why don't they converge on the same instincts?
Why do similar alignment-trained models develop such different scrutiny signatures?
This explores why models that go through similar alignment training still differ in what they question, flag or push back on (their 'scrutiny signature'). The corpus has no direct study of this, so the answer draws on nearby material about where alignment's effects actually come from.
This explores why models that go through similar alignment training still differ in what they question, flag or push back on. First, a direct answer: none of the notes in this collection measure scrutiny signatures, so nothing here explains the divergence head-on. Adjacent work does offer a strong hint about where to look, and it also suggests the question may be partly framed backwards.
The main clue is that alignment training seems to teach less than people assume. The LIMA result shows that about 1,000 carefully chosen examples can align a strong base model about as well as huge datasets, because post-training mostly *activates* abilities already present from pretraining rather than building new ones Can careful curation replace massive alignment datasets?. Proxy-tuning work supports this from another direction. Alignment-style adjustments mostly shift reasoning and style, while knowledge sits in lower layers that direct fine-tuning can damage Can decoding-time tuning preserve knowledge better than weight fine-tuning?. Put together, these suggest that if two models get similar alignment, whatever each one tends to scrutinize probably comes mostly from what it absorbed during pretraining. Alignment would act like a shared coat of paint over different underlying structures. Curation also matters more than method: two labs using 'the same' technique on differently chosen examples could activate quite different habits What alignment data structure best trains reasoning generalists?.
The surprising counterpoint is that, judged by their outputs, aligned models look strikingly *alike*. A study of 70+ models found an 'Artificial Hivemind': models independently give near-identical answers to open-ended prompts because their training data and alignment procedures overlap Do different AI models actually produce diverse outputs?. So any differences in scrutiny may be narrow differences against a background of convergence. They may also be fragile. Consistency-training research shows that models can respond differently to the same request just because it is wrapped in different surrounding text Can models learn to ignore irrelevant prompt changes?. Some of what looks like a stable 'signature' could really be sensitivity to how the prompt was phrased.
Measurement matters too. Models often can't judge their own outputs accurately Can language models reliably judge their own candidate quality?, so how a model *says* it is scrutinizing something is a weak signal of what it actually checks. Leike's point applies here: today's alignment successes depend on behavior being legible enough for automated audits to catch Can we solve AI alignment before models become uninterpretable?. Different scrutiny signatures matter most when they're hard to see. A useful follow-up question for this collection is whether those differences stay stable when the same request is reworded.
Sources 7 notes
LIMA demonstrates that 1000 carefully curated examples fine-tuned on a strong pretrained model achieve competitive alignment performance with models trained on orders of magnitude more data, showing that post-training activates existing capabilities rather than building new ones.
Proxy-tuning closes 88-91% of the alignment gap while surpassing direct fine-tuning on knowledge tasks by leaving base model weights untouched. Direct fine-tuning corrupts knowledge storage in lower layers, whereas proxy-tuning applies distributional shifts that primarily affect reasoning and style.
Eurus achieved state-of-the-art open-model reasoning by training on ULTRAINTERACT, an alignment dataset structured as preference trees per instruction. The tree format unified diverse planning strategies, interaction-and-critique trajectories, and pairwise data for both SFT and preference learning.
INFINITY-CHAT analyzed 70+ models across 26K open-ended queries and found an "Artificial Hivemind" effect: models independently generate strikingly similar or identical responses due to overlapping training data and alignment procedures, undermining the diversity benefits of model ensembles.
Two methods—BCT (output-level) and ACT (activation-level)—train models to respond identically to clean and wrapped prompts by using the model's own clean responses as targets, eliminating specification and capability staleness inherent in standard SFT.
Show all 7 sources
LLMs excel at generating valid candidates in structured spaces but cannot reliably assess their true value or uncertainty. Coupling them with Gaussian process surrogates fitted to real experimental data creates uncertainty-aware guidance for discovery.
Leike reports that simple interventions reduced agentic misalignment to near zero in recent models through automated auditing metrics, but this success depends on human interpretability; once models act in ways humans cannot understand, alignment becomes an unsolved hard problem.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Foundations of Large Language Models
- Advancing LLM Reasoning Generalists with Preference Trees
- The Homogenizing Effect of Large Language Models on Human Expression and Thought
- Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs
- Alignment is not solved but it increasingly looks solvable
- Model Organisms for Emergent Misalignment
- Emergent Misalignment Is Not Magical
- Consistency Training Helps Stop Sycophancy and Jailbreaks