INQUIRING LINE

Tell an AI a person's background and it still can't reliably guess how that particular person would answer a survey.

Can persona prompting improve prediction of individual survey responses?

This explores whether giving an LLM a person's profile or demographic traits (a persona prompt) helps it guess how that specific individual would answer a survey question, as opposed to how a group tends to answer.


This explores whether giving an LLM a person's profile or demographic traits helps it guess how that specific individual would answer a survey question, as opposed to how a group tends to answer. The corpus leans toward no. In the largest test it holds, conditioning models on participant profiles across 208,021 people didn't meaningfully improve predictions for specific individuals (Does conditioning LLMs on personal profiles improve prediction?). Persona prompting is the standard technique for making a model act like a particular person, and at the individual level it produces no measurable gain.

The mixed results people report for surveys seem to come from which questions get tested. Persona prompting does best on contested questions where humans disagree widely, and picking persona attributes using existing survey data on related questions improves the match with real answers (Why do persona prompts show such mixed results for surveys?). That fits a group-level reading. If a question splits the population along demographic lines, a persona can capture the split. It still doesn't tell you what one particular person will say.

Several other notes suggest why individual prediction stays out of reach. When the same persona prompt is run repeatedly, the variation between runs matches or exceeds the variation between different personas, so the model's own uncertainty is doing more work than any stable knowledge about the persona (Why do LLM persona prompts produce inconsistent outputs across runs?). Personas can also be worse than no persona. A plain 'how likely is a typical reader to click?' prompt ranked headlines better than a ten-persona demographic panel, which suggests the personas added systematic bias instead of audience insight (Do demographic personas help models rank headlines better?). And persona steering acts on the surface of the output. It follows the trait instruction but leaves the underlying bias in place (Can persona prompts actually reduce bias in language models?).

The positive results in the corpus come at an aggregate scale or from richer data. Personas built from real behavioral data predicted A/B test direction with 75 to 90 percent accuracy, but they were least reliable when the true effect was near zero, so they work for pre-screening and not for replacing live tests (Can behavior-based personas predict A/B test outcomes?). Simulated participants also replicated 76 percent of published marketing-experiment main effects, tracking how strong the original evidence was (Can AI personas reliably replicate human experiment results?). Both results are about what a population does on average, and both weaken as the signal gets small.

The most useful distinction is between predicting a crowd and predicting a person. Persona prompting looks like a reasonable tool for the first, especially on divisive questions and when the personas come from real data. It hasn't been shown to work for the second, and the corpus has no evidence that a demographic description can stand in for an individual's own answer. If you want individual-level forecasts, the evidence here points toward richer behavioral history and away from a more detailed persona paragraph.


Sources 7 notes

Does conditioning LLMs on personal profiles improve prediction?

Across 208,021 participants in the Psych-201 dataset, conditioning LLMs on participant profiles did not meaningfully improve predictions for specific individuals. The standard technique for individuation produces no measurable gains in person-level forecasting.

Why do persona prompts show such mixed results for surveys?

Survey simulation via persona prompting succeeds most on contested questions where humans disagree widely. Selecting persona attributes using existing survey data for related questions improves alignment between predicted and actual responses.

Why do LLM persona prompts produce inconsistent outputs across runs?

When the same persona prompt is run repeatedly, output variance across runs matches or exceeds variance across different personas. This reveals that model uncertainty, not stable social knowledge, drives persona-simulated outputs, making them unsuitable for simulating human annotation disagreement.

Do demographic personas help models rank headlines better?

A zero-shot baseline asking how likely a typical reader is to click ranked Upworthy headline variants significantly better than a ten-persona demographic panel. Persona conditioning appears to introduce systematic bias rather than capture audience-specific insight.

Can persona prompts actually reduce bias in language models?

Across three models, persona conditioning makes models follow trait instructions but fails to eliminate underlying bias. Between-group sentiment gaps persist unchanged, showing prompts operate only at the output level.

Show all 7 sources
Can behavior-based personas predict A/B test outcomes?

LLM agents conditioned on anonymized behavioral data predicted A/B test directions with 0.75–0.90 accuracy across 40 experiments. Predictions were most reliable for large effects and least trustworthy for near-zero effects, making the approach viable for fast pre-screening but not full replacement of live testing.

Can AI personas reliably replicate human experiment results?

Viewpoints AI reproduced 84 of 111 main effects from Journal of Marketing experiments with replication success strongly correlated to original p-value strength. Marginal effects showed unreliable performance with both false positives and negatives.

Papers this line draws on 8

The research behind the notes this line reads — ranked by how closely each paper relates.