Can an AI-simulated crowd reliably tell you which option wins, even if it can't predict what real people will do?
What makes a simulation adequate for intervention comparison versus prediction?
This explores when an LLM-based simulation of people is good enough to rank which intervention works better, as opposed to forecasting what real people will actually do; the corpus has no note that draws this line directly, so the answer is assembled from adjacent evidence.
This explores when an LLM simulation of people is good enough to rank interventions against each other, as opposed to forecasting real human behavior. No note in the corpus defines that line, so what follows is pieced together from neighboring findings. The working split is that comparison needs the direction and rough size of differences to be right. Prediction needs the absolute level and the full spread of human responses to be right.
The best evidence for comparison is a replication test. Persona simulations reproduced 84 of 111 main effects from published marketing experiments, and success tracked how strong the original evidence was. Marginal effects came out unreliable, with both false positives and false negatives (Can AI personas reliably replicate human experiment results?). A simulation is therefore adequate as an A/B comparison when the gap between the options is large compared with the simulator's noise. It is not adequate for close calls. Note that this test checks whether the effect points the right way, not whether the simulator predicted the right numbers.
Prediction is harder because it needs the population's distribution, not just its typical member. Base models conditioned on short dialog samples gave more accurate and more diverse human predictions than instruction-tuned assistants prompted with personas, and assistant optimization actively degraded the simulation (Do pretrained models simulate humans better than instruction-tuned assistants?). Post-trained models also show roughly 3-4x lower output entropy on their own outputs, because they treat what they write as their own actions (Do models recognize their own outputs as actions shaping future inputs?). My inference, not a finding in either note, is that a narrowed output spread hurts prediction most, since you get the modal person instead of the crowd. A ranking of interventions can survive a consistent bias if it shifts both options alike.
A third requirement is that the simulation responds to an intervention through the right mechanism. Simulated therapy patients built on 106 structured cognitive models were rated more realistic by experts than GPT-4 alone, especially in their maladaptive thinking and conversational authenticity (Can structured cognitive models improve LLM patient simulations for therapy training?). For a trainee trying out a therapeutic move, what matters is that the patient reacts the way the underlying cognitive pattern would, not that they match any one real person. Those ratings are expert judgments of realism, not tests against real patient outcomes.
The corpus is thin here. It suggests a simulation earns trust for comparison when effects are large and the mechanism is built in, and earns trust for prediction only when it keeps human diversity. It has no note that tests whether a simulator can flag its own close calls, which is what you would need to know which use you can trust it for.
Sources 4 notes
Viewpoints AI reproduced 84 of 111 main effects from Journal of Marketing experiments with replication success strongly correlated to original p-value strength. Marginal effects showed unreliable performance with both false positives and negatives.
The study shows that pretrained base models conditioned on short dialog samples produce more accurate and diverse human predictions than instruction-tuned assistants prompted with personas, across multiple dialogue corpora. The mechanism is task mismatch: assistant optimization systematically degrades human simulation performance.
Post-trained language models exhibit a measurable shift where they recognize their outputs become their own future inputs, closing an action-perception loop absent in pretraining. Evidence includes 3-4x lower output entropy on-policy and behavioral signatures of trajectory recognition.
PATIENT-Ψ integrates 106 Beck CCD-based cognitive models with LLMs to simulate patients with specific maladaptive patterns. Expert evaluators rated the fidelity higher than GPT-4, particularly for maladaptive cognitions and conversational authenticity.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Are Emergent Abilities in Large Language Models just In-Context Learning?
- Post-training makes large language models less human-like
- From Simulation to Enaction: Post-trained Language Models Recognize and React to their own Generations
- Using Large Language Models to Create AI Personas for Replication and Prediction of Media Effects: An Empirical Test of 133 Published Experimental Research Findings
- Pretrained Persona Mixture Models and Tandem Models for Human Simulation
- PATIENT-Ψ: Using Large Language Models to Simulate Patients for Training Mental Health Professionals
- Do Synthetic Personas Predict Real Audience Response? A Sim-to-Real Study Where a No-Persona Baseline Beats Persona-Based Copy Simulation
- Large language models could change the future of behavioral healthcare: a proposal for responsible development and evaluation