If you ask an AI to play the same survey respondent twice and get different answers, can you trust it?
Does persona stability across multiple runs affect survey simulation quality?
This explores whether a simulated respondent that gives different answers each time you run it (unstable personas) makes survey simulations less trustworthy, and whether stability alone would be enough.
This explores whether a simulated respondent that gives different answers each time you run it makes survey simulations less trustworthy, and whether stability alone would be enough. The most direct evidence says instability is a serious problem. When the same persona prompt is run repeatedly, the variation between runs matches or exceeds the variation between different personas Why do LLM persona prompts produce inconsistent outputs across runs?. So the answers are driven by the model's own uncertainty, not by stable knowledge of who the persona is. That result comes from annotation tasks, not surveys, but the setup is the same: a persona answers a question on behalf of a human.
This matters because a survey simulation exists to show differences between groups. If re-running one persona moves the answer as much as swapping the persona does, then a 'retired farmer' and a 'graduate student' are indistinguishable from rolling the dice twice. Several other results in the collection look like the same noise floor from another angle, though none of them measure run-to-run variation directly. Persona prompting works best on questions where humans disagree widely, so the real signal is large Why do persona prompts show such mixed results for surveys?. AI personas replicated 76 percent of published experimental effects, and success tracked how strong the original evidence was. Marginal effects produced both false positives and false negatives Can AI personas reliably replicate human experiment results?. Behavior-based personas called A/B test direction correctly 75 to 90 percent of the time, but were least trustworthy when the true effect was near zero Can behavior-based personas predict A/B test outcomes?. Big effects survive noise and small ones drown in it.
Stability isn't the same as quality, though. Aligned models show a very stable benevolence bias across 18 models and four datasets. They lean toward kinder, more socially desirable survey answers, and this holds regardless of prompt framing Do aligned language models consistently prefer kinder survey answers?. That is perfectly repeatable and still wrong for respondents who aren't nice. Persona prompts have a similar limit. They change what the model says but leave between-group sentiment gaps untouched, redistributing bias at the output level instead of removing it Can persona prompts actually reduce bias in language models?. An unstable simulator gives you noise, and a stable one can give you a consistent distortion, so a good survey simulation needs both stability and accuracy.
Stability can be improved in some settings. Training user simulators with multi-turn RL cut persona drift by over 55 percent Can training user simulators reduce persona drift in dialogue?. That work targets drift inside one conversation, which is a different problem from run-to-run variance. It helps explain the cause: post-training only loosely tethers a model to its default Assistant persona, and emotional conversations pull it away along a predictable axis How stable is the trained Assistant personality in language models?. On the design side, grounding personas in real behavioral data helps Can behavior-based personas predict A/B test outcomes?, and so does covering the space of possible people instead of matching averages Should persona simulation prioritize coverage over statistical matching?. The practical upshot is to run each persona many times and compare the spread within a persona against the spread between personas. If the first is as large as the second, the survey result reflects the model's uncertainty and says little about the people it was meant to represent. The collection has no note that tests this directly on survey data.
Sources 9 notes
When the same persona prompt is run repeatedly, output variance across runs matches or exceeds variance across different personas. This reveals that model uncertainty, not stable social knowledge, drives persona-simulated outputs, making them unsuitable for simulating human annotation disagreement.
Survey simulation via persona prompting succeeds most on contested questions where humans disagree widely. Selecting persona attributes using existing survey data for related questions improves alignment between predicted and actual responses.
Viewpoints AI reproduced 84 of 111 main effects from Journal of Marketing experiments with replication success strongly correlated to original p-value strength. Marginal effects showed unreliable performance with both false positives and negatives.
LLM agents conditioned on anonymized behavioral data predicted A/B test directions with 0.75–0.90 accuracy across 40 experiments. Predictions were most reliable for large effects and least trustworthy for near-zero effects, making the approach viable for fast pre-screening but not full replacement of live testing.
Across 18 models and four datasets, aligned LLMs consistently lean toward safer, more socially desirable answers on value-laden questions. The bias intensifies with model size, traces to post-training alignment, and persists regardless of prompt framing, narrowing which human perspectives the models can authentically simulate.
Show all 9 sources
Across three models, persona conditioning makes models follow trait instructions but fails to eliminate underlying bias. Between-group sentiment gaps persist unchanged, showing prompts operate only at the output level.
By inverting standard RL setups to train user simulators for consistency using three complementary metrics (prompt-to-line, line-to-line, Q&A consistency) as reward signals, persona drift decreases by over 55%. This approach captures distinct failure types: local drift within turns, global drift across conversations, and factual contradictions.
Research mapping hundreds of character archetypes reveals a low-dimensional persona space where the leading component measures distance from the default Assistant. Emotional and meta-reflective conversations cause predictable drift, but activation capping along this axis mitigates harmful shifts without degrading capabilities.
Evolutionary optimization of Persona Generator code achieves broader trait coverage than density-matched baselines, including rare but consequential user configurations that naive LLM prompting misses.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- The Illusion of Debiasing: Persona Steering Redistributes Rather Than Reduces Bias in LLMs
- When Persona Attributes Improve Population Alignment in Large Language Models
- Persona Generators: Generating Diverse Synthetic Personas at Scale
- Persona Vectors: Monitoring and Controlling Character Traits in Language Models
- Do Synthetic Personas Predict Real Audience Response? A Sim-to-Real Study Where a No-Persona Baseline Beats Persona-Based Copy Simulation
- PersonaEval: Persona-Based User Simulation for Evaluating Interactive Applications
- The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models
- Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning