Does alignment compound cultural bias that started during pretraining?
This explores whether the alignment/fine-tuning stage (RLHF, preference optimization) creates cultural bias or whether it inherits and concentrates bias already laid down during pretraining.
This explores whether alignment manufactures cultural bias or just inherits it — and the corpus points firmly at inheritance, with a twist: alignment rarely adds new bias, but it concentrates, hides, and entrenches what pretraining already planted. The cleanest causal result here is that cognitive and cultural biases are baked in during pretraining and only *swayed* by fine-tuning: models sharing a pretrained backbone show the same bias fingerprints no matter what instruction data they're tuned on Where do cognitive biases in language models come from?. So the source of the bias isn't alignment. The interesting question is what alignment then does to it.
The answer is that alignment compounds bias less by amplifying it and more by *masking* it. Indirect probes borrowed from psychology (Implicit Association Test–style methods) surface stereotypical associations that aligned models flatly refuse to state under direct questioning — alignment training conceals the bias rather than removing it Can indirect psychology tests reveal what LLMs conceal about bias?. That's a worse failure mode than raw bias, because the model now passes the surface audit while the underlying representation is unchanged. The cultural skew goes underground.
There's also a genuine narrowing mechanism, not just masking. When RL post-training runs, it converges on a single dominant distribution carried over from pretraining and collapses the alternatives — and which one wins depends on model scale, not on quality Does RL training collapse format diversity in pretrained models?. Read culturally, that's compounding: if pretraining over-represents one set of norms, alignment doesn't balance the field, it picks the majority channel and suppresses the rest. The same single-policy problem shows up in how alignment is even studied — the linguistic-alignment literature is built almost entirely on WEIRD (Western, educated, industrialized) samples, so a single alignment policy is unlikely to behave the same way across cultures Does linguistic alignment work the same way across cultures?.
The sharpest evidence that the bias is pretraining-deep, not alignment-deep, is the fingerprint of *shared* error. Across GPT-4.5, Gemini, and Claude, models beat almost every individual human at predicting social norms — yet they all make the *same* systematic mistakes on unwritten norms Can AI learn social norms better than humans? Can AI systems learn social norms without embodied experience?. Identical blind spots across independently aligned models means the gap lives in the shared substrate (web-scale pretraining text), and no amount of alignment closed it. Alignment also actively trims expression on top of this: RLHF rewards hedged, calibrated neutrality and structurally suppresses speech acts like alarm or warning Does alignment training suppress socially necessary speech acts? — a cultural flattening toward one rhetorical register.
The practical takeaway the reader may not have expected: the lever isn't more or better alignment, it's *where* you intervene. Decoding-time methods like proxy-tuning close most of the alignment gap while leaving the base weights — and the knowledge they store — untouched, whereas direct fine-tuning corrupts lower-layer storage Can decoding-time tuning preserve knowledge better than weight fine-tuning?. If bias is planted in pretraining and merely masked by alignment, then surface alignment is the wrong place to fix culture — you're papering over a foundation problem, and sometimes hiding it well enough that no one checks the foundation.
Sources 8 notes
A causal experiment using random-seed variation and cross-tuning showed that models sharing a pretrained backbone exhibit similar bias patterns regardless of finetuning data. Biases are planted during pretraining and merely swayed by instruction tuning.
Implicit Association Test-style probes reveal stereotypical associations in LLMs that the models refuse to report under direct questioning, showing that alignment training masks rather than eliminates underlying biases in representation.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
A 2020–2025 systematic review found that alignment effects are documented almost exclusively in WEIRD samples using inconsistent outcome measures, with mechanisms rarely directly measured. Communication norms vary substantially across cultures, making single alignment policies unlikely to produce uniform effects globally.
GPT-4.5 outperformed every individual human at judging social appropriateness across 555 scenarios, challenging the theory that embodied cultural experience is necessary. However, all AI models share identical systematic errors on unwritten norms.
GPT-4.5 predicted appropriateness of 555 social scenarios at the 100th percentile compared to human raters, with Gemini and Claude also exceeding 96% accuracy. However, all models show identical systematic errors, revealing boundaries of pattern-based social understanding that embodied experience may still be necessary to cross.
RLHF optimization rewards calibrated neutrality and hedged claims, which structurally prevents models from performing speech acts requiring overclaiming relative to baseline—like alarm, warning, prophecy, and denunciation. This is a direct consequence of the alignment objective, not a fixable bug.
Proxy-tuning closes 88-91% of the alignment gap while surpassing direct fine-tuning on knowledge tasks by leaving base model weights untouched. Direct fine-tuning corrupts knowledge storage in lower layers, whereas proxy-tuning applies distributional shifts that primarily affect reasoning and style.