Can you align an AI to written rules without collecting a single human preference label?

This explores whether alignment can skip the expensive human-labeling step by training a model to maximize the mutual information between a constitution and its responses, so that reading the principles makes the model's answers more predictable. The corpus says yes, at least in a striking proof of concept. The SAMI method fine-tunes a model to increase principle-response mutual information with no preference labels and no demonstrations, and a Mistral-7B trained this way beat both its base and instruction-tuned versions Can models learn behavioral principles without preference labels?. The most surprising twist: a weaker model could write the principles that aligned a stronger one — which flips the usual assumption that alignment supervision has to come from something more capable than the model being aligned.

Why this matters becomes clearer when you look at what preference labels actually carry. One line of work argues that annotation data isn't one clean signal at all — it decomposes into genuine preferences, non-attitudes, and constructed-on-the-spot preferences, and treating them uniformly quietly contaminates reward-model training Do all annotation responses measure the same underlying thing?. A related critique goes further: preferences don't capture thick moral values, and aggregating them produces systematic misalignment rather than agreement Should AI alignment target preferences or social role norms?. If the label itself is noisy or conceptually wrong, an approach that aligns to explicit written principles instead of aggregated votes is attractive — you get to state what you want rather than infer it from thousands of pairwise comparisons like those that power crowdsourced leaderboards Can crowdsourced votes reliably rank language models?.

SAMI belongs to a broader family of label-free methods that all lean on the model's own outputs as the training signal. Consistency training teaches a model to ignore irrelevant prompt wrapping by using its own clean responses as targets, sidestepping the staleness of hand-written SFT data entirely Can models learn to ignore irrelevant prompt changes?. Proxy-tuning shifts behavior at decoding time without touching base weights, preserving pretrained knowledge that direct fine-tuning tends to corrupt Can decoding-time tuning preserve knowledge better than weight fine-tuning?. The shared bet across all three: you can move a model toward desired behavior without a human sitting in the loop labeling which of two answers is better.

But the corpus also plants a warning worth carrying. Aligning to a constitution is still aligning to symbols, and one argument holds that symbolic goal encoding without world contact and social mediation can't guarantee the stated goals match real-world values — the words can drift from what they're supposed to mean Can AI systems achieve real alignment without world contact?. Maximizing mutual information makes responses faithful to the principles as written, not necessarily to what the principles were meant to protect. There's also a homogenization risk: alignment procedures already push different models toward eerily similar outputs — an 'artificial hivemind' — and optimizing everyone against similar constitutions could deepen that convergence Do different AI models actually produce diverse outputs?. So the honest answer is that mutual-information alignment can work without preference labels, and it neatly dodges the messiness of preference data — but it inherits a different, quieter problem: it makes a model loyal to the letter of its principles, which is not the same as loyal to their intent.