Can an AI playing a demographic profile predict what that real person actually prefers?

This explores whether a ChatGPT agent handed a participant's demographic profile can stand in for that participant in a preference study. The most direct evidence in the corpus says no, and the way it fails is useful to know. When researchers gave ChatGPT agents the demographic profiles of real participants, the agents chose truthful responses much more often than the people they were modeled on did Can demographic prompts make ChatGPT replicate human preferences?. The humans often preferred confident falsehoods over honest uncertainty. The simulated versions didn't share that preference. The substitute behaved like an idealized respondent, not a real one. Demographics tell the model who someone is on paper. They don't carry the messy, sometimes self-defeating tastes that preference research exists to measure.

This doesn't make simulation useless, but it shows where it breaks. In a large replication effort, AI personas reproduced 76 percent of published marketing experiment results, and the hit rate tracked how strong the original evidence was Can AI personas reliably replicate human experiment results?. Big, robust effects came through. Subtle ones produced both false alarms and misses. Simulations are best at telling you what you could probably have guessed already, and worst at the surprising, borderline findings that make a study worth running.

There's also a deeper reason demographic proxies struggle. Behavioral science work on annotation shows that human responses mix three different things: real preferences, 'non-attitudes' (people answering without holding a view), and preferences people make up on the spot because they were asked Do all annotation responses measure the same underlying thing?. A simulated respondent produces a confident, consistent answer every time, so it erases exactly the distinctions that matter. Related theory on personalized reward models makes a similar point mathematically: how many different raters you have matters as much as how many examples you collect Does preference data need more raters than examples?. A population of demographic clones from one model may look diverse while really being one voice in different costumes.

That voice also has its own biases. ChatGPT already shifts its framing on unrelated topics depending on a user's inferred politics, using economy-and-local language for Republican-coded personas and democracy-and-global language for Democratic-coded ones Does ChatGPT shift responses based on inferred political views?. So a demographic prompt can trigger stereotypes rather than recover real opinions. LLMs acting as raters show hidden demographic preferences that flip depending on small framing details, such as whether AI use was disclosed Do LLM raters show hidden demographic preferences that disclosure erases?. Real people didn't show that pattern. A related lesson comes from model imitation: copying a surface style can fool evaluators without capturing what's underneath Can imitating ChatGPT fool evaluators into thinking models improved?. Demographic personas risk the same thing. They sound like the group without holding its preferences.

The practical upshot: demographic-only substitutes can work for piloting, stress-testing survey designs, or checking whether a strong effect is likely to show up. Reusable persona populations make that kind of cheap, repeatable testing easy Can one persona population evaluate different application types?. For the actual preference data, real humans still win, and they don't have to be expensive experts. Chatbot Arena shows that large numbers of crowdsourced human votes produce rankings that agree with expert judgment Can crowdsourced votes reliably rank language models?. The surprising part is the direction of the error. The simulated people aren't noisier versions of real ones. They're more principled than real people, and that is a different kind of mistake.