AI stand-ins for survey participants nail the direction of big real-world effects, but stumble when the true effect is tiny.

This explores why AI personas standing in for human participants get the direction of a result right more often when the real-world effect is big, and what that means for when to trust them. The corpus documents the pattern from two independent directions. It never tests the mechanism directly, so the 'why' below is my inference from how the pieces fit.

The pattern first. Viewpoints AI reproduced 84 of 111 main effects from published marketing experiments, and its success tracked how strong the original result was. Marginal effects produced both false positives and false negatives Can AI personas reliably replicate human experiment results?. A separate study gave LLM agents real behavioral data and asked them to call the winner of 40 A/B tests. They were right 75 to 90 percent of the time, most reliable on large effects and least trustworthy on near-zero ones Can behavior-based personas predict A/B test outcomes?. Both studies score whether the simulation gets the direction right, not how big the effect is.

The likely reason is that a simulation carries a roughly constant amount of error, and a big effect is a signal loud enough to survive it. Persona prompts don't remove a model's biases. They redistribute them at the output level, and the gaps between groups stay put Can persona prompts actually reduce bias in language models?. Persona conditioning can even add noise. On Upworthy headlines, a plain 'how likely is a typical reader to click?' prompt ranked variants better than a ten-persona demographic panel Do demographic personas help models rank headlines better?. When the real effect is smaller than that background error, the simulation's answer is close to a coin flip that looks like a finding. That explains why near-zero effects are where both studies fall apart.

Big effects also tend to be broad ones, where most people react the same way. That kind of regularity is the part of human behavior a model absorbs in training, and it doesn't need the persona to capture anything about the individual. This fits a stubborn finding. Across 208,021 participants, conditioning LLMs on personal profiles did not meaningfully improve predictions for specific people Does conditioning LLMs on personal profiles improve prediction?. Persona simulations seem to work at the level of the average human, so they pick up effects strong enough to show at that level and miss the subtle, person-specific ones.

The practical upshot is that reliability belongs to the question being asked, not just to the tool. That makes personas good for pre-screening: run many ideas cheaply, trust the big directional calls, and send the marginal ones to live testing. The authors of the A/B study stop short of calling them a replacement for live tests Can behavior-based personas predict A/B test outcomes?. Two things suggest the weak zone won't vanish on its own. Persona consistency improved by only about 3 percent going from GPT 3.5 to Claude 3.5 Sonnet, despite a huge capability gap Does model capability translate to better persona consistency?. And a large average effect can hide rare but consequential users who react differently, which is why some researchers optimize persona sets to cover the edges instead of matching the crowd Should persona simulation prioritize coverage over statistical matching?.