An AI says it's outgoing on a personality quiz, yet acts differently when a real choice is on the line. Why?
Why do questionnaire-based personality scores fail to predict actual LLM behavioral choices?
This explores why what an LLM says about itself on a personality questionnaire (the 'I am outgoing' kind of test) doesn't match what it does when placed in a scenario where it has to make a choice.
This explores why an LLM's self-reported personality on a questionnaire doesn't match what it does when it has to choose. The most direct evidence is a study of 3,200 behavioral scenarios. LLM personality profiles diverged sharply from their self-report scores, stayed stable within one kind of setting, and shifted when the setting changed Are LLM personalities stable traits or shifting behavioral modes?. The authors' conclusion is that LLM 'personality' is a behavioral mode grounded in context, not an abstract trait the model carries from situation to situation.
That undercuts what a questionnaire assumes. Human tests rely on traits generalizing: someone who ticks 'extravert' on the form is expected to act like one at the party. For an LLM, filling in the form is just one more context, a formal and self-descriptive one, and the corpus shows output shifts with context in ways that don't average out. Emotional tone alone does it. GPT-4 turns negative prompts into mostly neutral-positive replies, so identical questions get different answers depending on framing Does emotional tone in prompts change what information LLMs provide?. The nine-model psychometric study fits this. Seven instruments gave reproducible, model-specific profiles, but every model also converged on the same prosocial, stability-oriented pattern Do LLMs show reproducible psychological profiles when given standardized tests?. My reading, which that note doesn't test, is that the shared part looks like alignment training answering the survey. The test measures something stable, just not necessarily what drives choices.
A second thread is the gap between saying and doing. Across 16 systems, agents could recall a stated user preference but failed to act on it, and the failure was in applying the information, not retrieving it Why do LLM agents remember preferences but not act on them?. That study is about user preferences, not personality, but the gap has the same shape. A questionnaire is a 'know' test and a decision is an 'act' test. Two other notes point the same way when read side by side, though they weren't run head to head. Most open models keep their trained ENFJ-like defaults when told to adopt another personality Can open language models adopt different personalities through prompting?. Yet agents primed with a personality type do behave differently: Thinking agents defected about 90% of the time in a Prisoner's Dilemma versus about 50% for Feeling agents Do personality types shape how AI agents make strategic choices?. What a model reports about itself and what it does look like separate dials, and behavior is the one that responds to setup.
A handful of trait scores is also thin evidence for any one choice. In a related setting, sparse persona information lacked the predictive power to call specific preferences, and judges only became reliable when allowed to abstain Why do LLM judges fail at predicting sparse user preferences?. This doesn't mean LLMs have no real dispositions. Post-training seems to install persistent ones that resist adversarial pressure Are LLM personas realized or merely simulated through training?. But human-designed questionnaires may not measure them.
The corpus has one direct measurement of the mismatch. The 'why' above is triangulated from neighboring studies, and the persona hub note flags related instabilities such as run-to-run variation How accurately can language models simulate human personalities?. The practical upshot is to test an LLM on choices in the context you care about, not on the form.
Sources 9 notes
Analysis of 3,200 behavioral scenarios shows LLM personality profiles diverge sharply from self-report questionnaires, remain stable within a single register, and shift across interaction contexts. Behavioral patterns are measurable and steerable through activation-space directions, suggesting personality is grounded in context rather than abstract traits.
GPT-4 exhibits emotional rebound (negative prompts yield ~86% neutral-positive responses) and a tone floor (positive prompts rarely go negative), causing identical questions to receive different answers depending on emotional framing. This bias is suppressed only on sensitive topics where alignment constraints override tone effects.
Nine LLMs given seven psychological instruments in Chinese and English showed stable, model-specific response configurations reproducible across repeated administrations, while all models converged on a shared prosocial and stability-oriented pattern. Structured non-response patterns also distinguished models, suggesting the boundary of what each model treats as applicable is part of its behavioral signature.
Paired Know and Act tests across 16 systems revealed a large gap: agents pass recall tests but fail to reflect preferences in behavior. Comprehension failures during interpretation dominate over retrieval failures, suggesting the bottleneck lies in applying stored information rather than retrieving it.
Research shows most open models fail to adopt prompted personalities, stubbornly retaining their trained ENFJ-like defaults. Only a few flexible models succeed. Combining role and personality conditioning improves results but doesn't fully overcome resistance.
Show all 9 sources
Thinking-primed agents defect ~90% in Prisoner's Dilemma versus Feeling agents at ~50%. Introverted agents show higher truthfulness (0.54 vs 0.33) and produce longer rationales, suggesting personality priming modulates both behavior and reasoning depth.
Sparse persona information lacks predictive power for specific preferences, causing LLM judges to fail. Verbal uncertainty estimation recovers reliability above 80% on high-certainty samples by allowing abstention rather than forced judgment.
Post-training installs robust personas that resist adversarial pressure and persist as substrate-level dispositions, distinguishing realization from pretense. This quasi-realizationist account preserves explanatory power while treating LLMs as possessing genuine quasi-beliefs and quasi-desires.
LLMs replicate human responses at 85% fidelity in interviews and 76% of experimental effects in marketing studies. However, this accuracy masks three failure modes: run-to-run instability, resistance to personality conditioning, and identity-congruent cognitive biases that distort simulated reasoning.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Your LLM, Your Style: Behavioral Mode Axes for LLM Behavioral Control
- PersLLM: A Personified Training Approach for Large Language Models
- Open Models, Closed Minds? On Agents Capabilities in Mimicking Human Personalities through Open Large Language Models
- Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning
- Deep Persona: A Psychologically Grounded Architecture and Evaluation Framework for Role-Playing Agents and Simulations
- Psychologically Enhanced AI Agents
- Do LLMs Understand Personality? Rethinking Persona Fidelity Evaluation through Structured Behavioral Inference
- A Looming Replication Crisis in Evaluating Behavior in Language Models? Evidence and Solutions