Data-Driven Persona-Conditioned Agents for A/B Test Simulation

A/B testing is the gold standard for evaluating product changes, but each experiment requires real user traffic, engineering effort, and weeks of measurement. We propose a simulation framework that predicts A/B test outcomes using LLM-powered agents conditioned on data-driven personas grounded in real user behavioral signals. Unlike prior work that relies on synthetic or rule-based personas, our agents are constructed from anonymized behavioral data—activity patterns, engagement signals, and inferred demographics—enabling more faithful population modeling. We frame A/B test simulation as a structured question task and systematically study (i) question design formats, (ii) the impact of persona data source and domain alignment, (iii) the trade-off between per-persona behavioral depth and population diversity, and (iv) efficient population subsampling. On a benchmark of 40 A/B tests spanning two metric types, our best configuration achieves 0.75–0.90 directional accuracy depending on the test metric, demonstrating that data-driven personas are a viable path toward fast, low-cost experiment pre-screening.

Introduction. LLMs can emulate human decision-making across tasks from survey response prediction (Argyle et al., 2023; Aher et al., 2023) to preference elicitation and behavioral role-play (Wang et al., 2025; Mansour et al., 2025). When conditioned on detailed user profiles, LLM-powered agents approximate the judgments of specific population segments, enabling scalable, low-cost simulation of collective user behavior (Park et al., 2024; Bui et al., 2025). A particularly compelling application is the simulation of online controlled experiments (A/B tests): if persona-conditioned agents can reliably predict whether users prefer a treatment variant over a control, teams could pre-screen design candidates offline—reducing the time, traffic, and experimentation cost (Rieder et al., 2026; Castelo et al., 2026). Online controlled experiments remain the gold standard for validating product changes, yet each test requires sufficient user traffic, engineering effort, and typically weeks of data collection to reach statistical significance (Kohavi et al., 2009).

Discussion / Conclusion. Potential applications. With current accuracy levels, the proposed framework cannot fully replace human A/B tests—but it does not need to. A potential application could be a pre-screening tool that filters clearly inferior treatment candidates before they consume traffic and prioritizes the experiments by ranking proposed changes by predicted impact. With batch inference, a full simulation could complete in hours at a fraction of a multiweek experiment cost, potentially enabling teams to explore a broader design space without proportionally increasing experimentation overhead. When to trust simulation. Our experiments show that simulations are most reliable when the underlying effect size is large, as is typical of highsalience decisions. Conversely, the framework is least trustworthy for near-zero effects where small Data requirements. Domain alignment matters more than data volume or source exclusivity. Public e-commerce data rivals platform-specific personas (Table 2), lowering adoption barriers.