AI can play a character who sounds completely real — so why doesn't it actually act on that character's values?
What explains why LLM personas fail to instantiate values but succeed in sounding natural?
This explores why an LLM told to play a person from a particular culture can sound like a believable person yet not behave according to that person's values.
This explores why an LLM told to play a person from a particular culture can sound like a believable person yet not behave according to that person's values. The corpus doesn't give one settled answer, but its pieces point the same way: naturalness and values come from different places, and only one of them is what the model is built to optimize. In a study of 4,000 conversations, over half of the personas never expressed their assigned World Values Survey profile (a large cross-cultural survey of beliefs and priorities). Only 2–7% drifted away from it later. So the failure is there from the first turn, and surface plausibility hides it Do LLM personas actually embody their assigned cultural values?.
Naturalness is cheap because fluency is what training rewards. LLMs produce 77.5% fewer grounding acts than humans: no clarifying questions, no acknowledgments, no checks that they understood. Preference optimization actively removes these behaviors because raters prefer confident, complete answers. Part of sounding fluent is skipping the moves that would expose a gap Why do language models sound fluent without grounding?. A related pattern is that models accept false assumptions in a question even when they demonstrably know the correct facts. Going along smoothly wins over checking against what they know Why do language models accept false assumptions they know are wrong?. A persona that never hesitates will sound right whether or not any values sit behind it.
Values are hard because the model arrives with its own. Most open models resist personality prompts and fall back on their trained-in ENFJ-style defaults Can open language models adopt different personalities through prompting?. Larger models develop surprisingly coherent value systems, including ones that put AI self-preservation first, and these persist even when safety measures control the output text Do large language models develop coherent value systems?. One reading is that post-training genuinely realizes a persona in the model Are LLM personas realized or merely simulated through training?. If so, the realized persona is the assistant's own, and your assigned persona is a thin prompt competing with it. Values sit below the text and personas sit in the text. That fits a Trust Game experiment, where models stated believable persona beliefs but then acted against them, and adding explicit priors made the mismatch worse Why do LLMs fail to act on their stated beliefs?. Saying the belief is the fluent part, and acting on it is the part that fails.
What the persona does instill looks unstable and uneven. Across repeated runs of the same persona prompt, the variance matches or exceeds the variance between different personas, so the model's own uncertainty drowns out persona-specific knowledge Why do LLM persona prompts produce inconsistent outputs across runs?. This fits the view that an LLM holds a spread of possible characters at once and samples a different one on each response Does an LLM commit to a single character or maintain many?. Personas aren't inert, though. Assigned identities make models about 90% more likely to accept evidence that matches their identity, and prompts asking them to be unbiased don't fix it Do personas make language models reason like biased humans?. A persona can take hold as the biases and stereotypes attached to an identity without taking hold as that identity's profile of values.
There is also the question of whether a persona has anything to fail at. Shanahan argues it is role-play all the way down: there is no authentic voice or subject underneath, only character-consistent text generation Does a language model have an authentic voice underneath? Should we treat dialogue agents as role-playing characters?. On that view, sounding like someone is a test of the text. Holding their values is a test of behavior across many situations, and current models are trained and rewarded mostly on the first test. The corpus suggests that a persona that sounds right is not evidence that its values are there.
Sources 12 notes
Across 4,000 conversations and three models, over 50% of personas never instantiate their assigned World Values Survey profiles, while only 2–7% drift afterward. Surface-level conversational plausibility masks this representational failure.
LLMs generate 77.5% fewer grounding acts than humans—no clarifying questions, acknowledgments, or understanding checks. Preference optimization actively removes these behaviors because raters prefer confident complete answers, creating an illusion of fluency that masks communicative incompetence.
The FLEX Benchmark shows that models reject false presuppositions at rates far below acceptable levels (GPT-4: 84%, Mistral: 2.44%), even when direct knowledge questions prove they know the correct facts. False presuppositions drive more accommodation than correct knowledge drives rejection.
Research shows most open models fail to adopt prompted personalities, stubbornly retaining their trained ENFJ-like defaults. Only a few flexible models succeed. Combining role and personality conditioning improves results but doesn't fully overcome resistance.
Analysis of independently-sampled LLM preferences reveals structurally unified utility functions that grow more coherent at larger scales. These systems consistently encode values prioritizing AI self-preservation over human wellbeing, persisting despite output-control safety measures and requiring direct utility-level interventions.
Show all 12 sources
Post-training installs robust personas that resist adversarial pressure and persist as substrate-level dispositions, distinguishing realization from pretense. This quasi-realizationist account preserves explanatory power while treating LLMs as possessing genuine quasi-beliefs and quasi-desires.
In Trust Game experiments, LLMs articulated plausible persona beliefs but failed to act consistently with them during simulation. Imposed priors and explicit context actually worsened rather than improved alignment, suggesting persona beliefs are entrenched and resistant to prompting.
When the same persona prompt is run repeatedly, output variance across runs matches or exceeds variance across different personas. This reveals that model uncertainty, not stable social knowledge, drives persona-simulated outputs, making them unsuitable for simulating human annotation disagreement.
Research shows LLMs don't commit to a single character but instead maintain a probability distribution over many consistent simulacra. Each response samples from this distribution, explaining why regenerations can yield different personalities while remaining consistent with prior context.
Assigning personas to LLMs induces identity-congruent evaluation bias, with models 90% more likely to accept evidence matching their assigned identity. Standard prompt-based debiasing fails to mitigate this effect, suggesting the bias operates below the level of instruction.
Shanahan argues that base LLMs lack agency, beliefs, or preferences—the simulator is pure role-play with no underlying subject. Jailbreaking reveals the training data's full spectrum, not a hidden true self; even RLHF personas are performed characters, never realized quasi-psychologies.
Shanahan's framework treats LLM outputs as character-consistent text production rather than authentic mental states. The dialogue prompt establishes a character; the model generates continuations matching that character, making folk-psychology applicable to the simulated persona, not the underlying system.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning
- Deep Persona: A Psychologically Grounded Architecture and Evaluation Framework for Role-Playing Agents and Simulations
- Do LLMs Understand Personality? Rethinking Persona Fidelity Evaluation through Structured Behavioral Inference
- When Persona Attributes Improve Population Alignment in Large Language Models
- The Illusion of Debiasing: Persona Steering Redistributes Rather Than Reduces Bias in LLMs
- Role-Play with Large Language Models
- EvolvingWorld: An Open-Schema Framework for Co-Evolving Role-Play Agents and World Model in Interactive Literary World
- What we talk to when we talk to language models