Tell an AI to play a role, then tell it to 'be fair' — does the fairness instruction actually work?
Can debiasing instructions override bias introduced by persona assignment?
This explores whether telling a model to 'be unbiased' can cancel out the skew that appears once you give it a persona or identity to play.
This explores whether telling a model to 'be unbiased' can cancel out the skew that appears once you give it a persona. The corpus's answer is mostly no, at least for plain prompt-level instructions. The most direct evidence is a study where assigning personas made models reason like motivated humans: they were about 90% more likely to accept evidence that matched their assigned identity. Standard prompt-based debiasing failed to fix it, which suggests the bias sits below the level an instruction can reach (Do personas make language models reason like biased humans?).
A second study points the same way from the opposite direction. Persona prompts do change what models say: they follow the trait instructions. But the between-group sentiment gaps stayed the same. The prompt moves bias around in the output without shrinking it, so it operates in the 'output channel' and doesn't touch the underlying tendency (Can persona prompts actually reduce bias in language models?). A debiasing instruction is just another prompt in that same channel. That may be why these instructions feel as if they should work: the model visibly obeys the persona and visibly obeys the caveat, while the underlying skew stays put.
The corpus also suggests why the instruction lands so weakly. Cognitive biases in language models are mostly planted during pretraining, and finetuning only modulates them. Models that share a pretrained backbone show similar bias patterns whatever they are finetuned on (Where do cognitive biases in language models come from?). If instruction tuning barely reshapes a bias, one sentence of instruction at inference time is unlikely to. Persona effects are also noisy. Repeated runs of the same persona vary as much as different personas do, so model uncertainty is a big part of what the persona 'says' (Why do LLM persona prompts produce inconsistent outputs across runs?). A debiasing instruction is trying to steer something that is only partly stable in the first place.
What the corpus offers instead are training-time fixes, and it does not test them on persona bias directly. Consistency training teaches a model to answer the same way whether or not the prompt is wrapped in irrelevant extras, using its own clean responses as targets (Can models learn to ignore irrelevant prompt changes?). Counterfactual invariance in reward modeling forces predictions to stay stable when irrelevant variables change, and it removed four reward-hacking biases, including sycophancy and discrimination (Can counterfactual invariance eliminate reward hacking biases?). Both treat a persona as an irrelevant variable the answer should not depend on. That is a plausible next step, but the notes here don't show it working for persona-induced bias. What they show is that asking nicely doesn't.
Sources 6 notes
Assigning personas to LLMs induces identity-congruent evaluation bias, with models 90% more likely to accept evidence matching their assigned identity. Standard prompt-based debiasing fails to mitigate this effect, suggesting the bias operates below the level of instruction.
Across three models, persona conditioning makes models follow trait instructions but fails to eliminate underlying bias. Between-group sentiment gaps persist unchanged, showing prompts operate only at the output level.
A causal experiment using random-seed variation and cross-tuning showed that models sharing a pretrained backbone exhibit similar bias patterns regardless of finetuning data. Biases are planted during pretraining and merely swayed by instruction tuning.
When the same persona prompt is run repeatedly, output variance across runs matches or exceeds variance across different personas. This reveals that model uncertainty, not stable social knowledge, drives persona-simulated outputs, making them unsuitable for simulating human annotation disagreement.
Two methods—BCT (output-level) and ACT (activation-level)—train models to respond identically to clean and wrapped prompts by using the model's own clean responses as targets, eliminating specification and capability staleness inherent in standard SFT.
Show all 6 sources
Causal reward modeling using counterfactual invariance constrains reward predictions to remain consistent when irrelevant variables change, eliminating length bias, sycophancy bias, concept bias, and discrimination. Standard training cannot distinguish causal from spurious features; counterfactual invariance forces isolation of actual quality signals.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- The Illusion of Debiasing: Persona Steering Redistributes Rather Than Reduces Bias in LLMs
- When Persona Attributes Improve Population Alignment in Large Language Models
- Persona-Assigned Large Language Models Exhibit Human-Like Motivated Reasoning
- Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization
- Are Emergent Abilities in Large Language Models just In-Context Learning?
- Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking
- Planted in Pretraining, Swayed by Finetuning: A Case Study on the Origins of Cognitive Biases in LLMs
- Consistency Training Helps Stop Sycophancy and Jailbreaks