Can AI 'users' built from free, public material stand in for real customers, or do you need private data?

This explores whether personas built from openly available material (published documents, existing persona datasets, public research) can do what personas built from private, real-user behavioral data do. The corpus has no head-to-head test with the same task and both data types. The pieces it does have suggest a split: public material works well for personas that represent perspectives, and less well for personas that must predict what real people will do.

The public-material side is stronger than you might expect. Can one persona population evaluate different application types? shows that one population of simulated users from existing persona datasets can evaluate surveys, chatbots and other applications through plug-and-play adapters, with no rebuilding per task. Can personas extracted from documents generalize across evaluation tasks? goes further and pulls personas straight out of domain documents by clustering stakeholder viewpoints. The resulting judges transfer between summarization and dialogue without redesign, because they are grounded in real perspectives rather than arbitrary roles. Can synthetic dialogues become realistic through layered diversity? finds that layering subtopic, Big Five personality and context gets synthetic dialogue to about 90% of in-domain performance, with no private user logs involved.

Behavior-based personas are the proprietary-style benchmark. Can behavior-based personas predict A/B test outcomes? conditioned LLM agents on anonymized behavioral data and predicted the direction of 40 A/B tests with 75 to 90 percent accuracy. Accuracy was highest for big effects and lowest for near-zero ones, so it works as a pre-screen and not a replacement for live testing. Can AI personas reliably replicate human experiment results? shows the same shape from a different angle. A persona simulator matched 84 of 111 published marketing-experiment effects, closely tracking how strong the original evidence was, and was unreliable on marginal effects. The ceiling in both cases is effect size, not where the data came from. That suggests data provenance may not be the main bottleneck.

The sharpest constraint is in How do we generate realistic personas at population scale?. Persona generation at scale produces systematic bias, for example in election forecasting, because the heuristics used cannot recover the true joint distribution from marginal data. My reading is that this is roughly the gap between published summary statistics (averages for each trait separately) and person-level records (how traits combine in one individual). If so, that is where public data loses to proprietary data. The note's proposed fix is a public one: ImageNet-style benchmarks and training datasets for personas. In other words, the field's answer to this gap is to build the open resource.

So public data looks competitive for reusable evaluation, perspective-taking and diverse synthetic conversations. Proprietary behavioral data has the edge when you need to predict specific real users' choices, and even then it only reliably catches large effects. The open question the corpus leaves is whether a well-built public joint-distribution dataset would close that gap.