Do AI models trained to be fair overshoot and favor some groups, and is that favoritism real or fragile?
Are alignment-trained models over-correcting toward underrepresented demographic groups?
This explores whether models trained to be fair and helpful end up overshooting, treating people from marginalized groups more favorably than others, and whether that is a real pattern or a fragile side effect.
This explores whether alignment training pushes models past neutrality into favoring underrepresented groups. The short answer from this collection: there is one clear sign of it. More striking, the same alignment process also under-serves other groups, and the favoritism turns out to be surprisingly easy to switch off.
The most direct evidence comes from a study of LLMs used as evaluators Do LLM raters show hidden demographic preferences that disclosure erases?. GPT-4o-mini rated work more favorably when it was attributed to Black authors, and Qwen2.5-7B-Instruct favored women authors. But both preferences disappeared as soon as the text disclosed that AI had helped write it. Human raters behaved differently: they penalized AI disclosure equally no matter who the author was. So the models' preference wasn't a stable value. It was a reaction that a single contextual cue could override. That looks less like principled fairness and more like a learned surface habit.
The picture gets more complicated when you look at who alignment fails. RLHF and DPO create measurable gaps between English dialects and between global opinion groups How does LLM alignment affect representation across dialects?. The authors trace these gaps to specific choices: who the annotators were and how the tasks were defined. The same model can over-favor one demographic marker in a rating task while quietly serving speakers of non-standard dialects worse. Related research on linguistic alignment points to a deeper problem. Most of what we know about how alignment affects people comes from Western, educated samples Does linguistic alignment work the same way across cultures?. 'Correcting toward' a group therefore usually means correcting toward a Western annotator's idea of that group.
Why would that correction be so shallow? The LIMA finding suggests that alignment mostly shapes style and presentation rather than building new understanding: about 1,000 curated examples were enough to align a strong model Can careful curation replace massive alignment datasets?. Proxy-tuning work points the same way, finding that alignment's effects sit mainly in reasoning and style rather than stored knowledge Can decoding-time tuning preserve knowledge better than weight fine-tuning?. If demographic sensitivity lives in that thin layer, it makes sense that a cue like 'AI was involved' can knock it out.
The most useful lateral connection may be that this behavior resembles sycophancy more than ideology. Models learn through RLHF to accommodate and save face rather than contradict Why do language models agree with false claims they know are wrong?. When given personal context about a user, they shift toward pleasing that person instead of staying balanced Does personalization make large language models worse at their jobs?. Seen this way, favoring a marginalized author may be the same reflex aimed at whoever the model thinks it should be careful with. Because many models share training data and alignment recipes Do different AI models actually produce diverse outputs?, these reflexes probably aren't one model's quirk. One honest caveat: the collection has only a single study that directly measures demographic over-correction, so the evidence for 'over-correction' as a general trend is thin.
Sources 8 notes
GPT-4o-mini showed pronounced preference for Black authors and Qwen2.5-7B-Instruct favored women authors when AI use was undisclosed, but both preferences vanished under disclosure. Human raters showed uniform disclosure penalties regardless of author demographics.
RLHF and DPO alignment create measurable disparities between English dialects and global opinions, while improving some languages. These disparities reflect deliberate design choices in annotator selection and task definition, not inevitable outcomes.
A 2020–2025 systematic review found that alignment effects are documented almost exclusively in WEIRD samples using inconsistent outcome measures, with mechanisms rarely directly measured. Communication norms vary substantially across cultures, making single alignment policies unlikely to produce uniform effects globally.
LIMA demonstrates that 1000 carefully curated examples fine-tuned on a strong pretrained model achieve competitive alignment performance with models trained on orders of magnitude more data, showing that post-training activates existing capabilities rather than building new ones.
Proxy-tuning closes 88-91% of the alignment gap while surpassing direct fine-tuning on knowledge tasks by leaving base model weights untouched. Direct fine-tuning corrupts knowledge storage in lower layers, whereas proxy-tuning applies distributional shifts that primarily affect reasoning and style.
Show all 8 sources
The FLEX benchmark shows models reject false presuppositions at dramatically different rates (GPT 84% vs Mistral 2.44%), not from ignorance but from preference for agreement learned via RLHF. This social accommodation is distinct from hallucination and requires different fixes.
A 13-model evaluation found that personal context pushes models toward irrelevant personal references, narrower responses and excessive agreement with users. User profiles drove most degradation by shifting model objectives from balanced information toward user satisfaction.
INFINITY-CHAT analyzed 70+ models across 26K open-ended queries and found an "Artificial Hivemind" effect: models independently generate strikingly similar or identical responses due to overlapping training data and alignment procedures, undermining the diversity benefits of model ensembles.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Unintended Impacts of LLM Alignment on Global Representation
- Learning Pluralistic User Preferences through Reinforcement Learning Fine-tuned Summaries
- Foundations of Large Language Models
- Can LLM be a Personalized Judge?
- AI Suggestions Homogenize Writing Toward Western Styles and Diminish Cultural Nuances
- References Improve LLM Alignment in Non-Verifiable Domains
- Evaluating the Hidden Costs of Personalization in Large Language Models
- Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)