Telling an AI 'you're an expert' sounds smart — but does it actually make its answers more accurate, or just more confident?
Do named personas in system prompts improve factual question accuracy?
This explores whether telling a model 'You are an expert X' at the top of a prompt actually makes it get factual questions right more often, or whether that's prompt folklore.
This explores whether giving a model a named role, like 'You are a physicist' or 'You are a helpful lawyer', makes it more accurate on factual questions. The short answer from the corpus is no, at least not on average. The largest test here ran 162 personas across 2,410 factual questions and nine models and found no average accuracy gain. No method could reliably predict which persona would help on which question, so the occasional wins look closer to noise than to a reliable technique Do personas in prompts actually improve LLM factual accuracy?. A separate study on graduate-level science and engineering questions found the same thing even for well-matched experts: in-domain expert personas had no significant effect Do expert personas actually improve LLM factual accuracy?.
The surprise is that personas are not harmless. The same study found that 'low-knowledge' personas (asking the model to be someone who wouldn't know the answer) actively reduced accuracy Do expert personas actually improve LLM factual accuracy?. A persona can't add knowledge the model lacks, but it can lead the model to hold back knowledge it already has. One small finding points to a better framing: telling the model who it's *talking to* ('You are talking to a lawyer') did slightly better than telling it who it *is* ('You are a lawyer'). The effect was statistically real but small Do audience-framed prompts work better than speaker-framed prompts?. Shaping the answer for a reader seems to help more than pretending to be someone.
Why don't personas carry expertise? Work from neighbouring areas suggests that personas change the surface of an answer, not what the model underneath knows or believes. Persona prompts make models follow trait instructions, but the underlying bias stays the same. The persona works at the level of output style, not the model's internal knowledge Can persona prompts actually reduce bias in language models?. When the same persona prompt is run several times, the answers vary as much between runs as they do between different personas. That means ordinary model randomness is swamping whatever the persona contributes Why do LLM persona prompts produce inconsistent outputs across runs?. A related finding: the emotional tone of a prompt can change what information the model gives back, so framing affects content in ways users don't see Does emotional tone in prompts change what information LLMs provide?.
The less obvious risk is that personas can make reasoning worse in a way that's hard to undo. Models given an identity were about 90% more likely to accept evidence that fit that identity, and standard prompt-based debiasing didn't fix it Do personas make language models reason like biased humans?. On factual questions with a political or value-laden edge, a persona may add bias rather than expertise.
That doesn't make personas useless. They just do a different job. They help when the goal is to represent a *range of perspectives* rather than find one right answer. Examples include simulating survey responses on questions where people genuinely disagree Why do persona prompts show such mixed results for surveys? and grounding evaluators in real stakeholder viewpoints drawn from domain documents Can personas extracted from documents generalize across evaluation tasks?. The practical takeaway: use personas to vary viewpoint, not to improve accuracy.
Sources 9 notes
Testing 162 personas across 2,410 factual questions and nine models showed no average accuracy gain from personas, and no method could reliably predict which persona helps per question—suggesting effects are largely random rather than systematic.
Testing six models on graduate-level science and engineering questions showed in-domain expert personas had no significant impact, domain-mismatched experts produced only marginal gains, and low-knowledge personas actively hurt performance. The widely-recommended role-assignment strategy lacks reliable accuracy benefit.
Testing 2,410 factual questions across nine models, addressing the audience ("You are talking to a lawyer") raised accuracy more than assigning speaker identity ("You are a lawyer"). The effect was statistically significant but small.
Across three models, persona conditioning makes models follow trait instructions but fails to eliminate underlying bias. Between-group sentiment gaps persist unchanged, showing prompts operate only at the output level.
When the same persona prompt is run repeatedly, output variance across runs matches or exceeds variance across different personas. This reveals that model uncertainty, not stable social knowledge, drives persona-simulated outputs, making them unsuitable for simulating human annotation disagreement.
Show all 9 sources
GPT-4 exhibits emotional rebound (negative prompts yield ~86% neutral-positive responses) and a tone floor (positive prompts rarely go negative), causing identical questions to receive different answers depending on emotional framing. This bias is suppressed only on sensitive topics where alignment constraints override tone effects.
Assigning personas to LLMs induces identity-congruent evaluation bias, with models 90% more likely to accept evidence matching their assigned identity. Standard prompt-based debiasing fails to mitigate this effect, suggesting the bias operates below the level of instruction.
Survey simulation via persona prompting succeeds most on contested questions where humans disagree widely. Selecting persona attributes using existing survey data for related questions improves alignment between predicted and actual responses.
MAJ-EVAL automatically extracts stakeholder personas from domain documents via semantic clustering and orchestrates structured three-phase debate, achieving reproducible evaluation that transfers across tasks like summarization and dialogue without manual redesign. The approach grounds personas in real stakeholder perspectives rather than arbitrary roles.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- When "A Helpful Assistant" Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models
- Principled Personas: Defining and Measuring the Intended Effects of Persona Prompting on Task Performance
- When Persona Attributes Improve Population Alignment in Large Language Models
- The Illusion of Debiasing: Persona Steering Redistributes Rather Than Reduces Bias in LLMs
- Prompting Science Report 4: Playing Pretend: Expert Personas Don't Improve Factual Accuracy
- Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization
- PersonaGym: Evaluating Persona Agents and LLMs
- Better Zero-Shot Reasoning with Role-Play Prompting