Can behavior-based personas predict A/B test outcomes?

The paper proposes a simulation framework that predicts A/B test outcomes with LLM agents conditioned on personas built from "anonymized behavioral data—activity patterns, engagement signals, and inferred demographics," in contrast to "prior work that relies on synthetic or rule-based personas." On a benchmark of 40 A/B tests spanning two metric types, the best configuration reaches "0.75–0.90 directional accuracy depending on the test metric." The authors call this "a viable path toward fast, low-cost experiment pre-screening" and concede that at current accuracy the framework "cannot fully replace human A/B tests—but it does not need to."

The task is framed as a "structured question task," and the abstract lists four design questions: question format, persona data source and domain alignment, the trade-off between per-persona behavioral depth and population diversity, and efficient population subsampling. The discussion gives two conclusions. Simulations are "most reliable when the underlying effect size is large, as is typical of high-salience decisions," and least trustworthy for near-zero effects. And "domain alignment matters more than data volume or source exclusivity": public e-commerce data "rivals platform-specific personas," which lowers the barrier to adopting the method.

This is the prospective, product-side counterpart to Can AI personas reliably replicate human experiment results?. That study replicated published effects and found success tracking evidence strength; this one predicts new treatment-versus-control outcomes and reports the same shape, with large effects reliable and near-zero ones not. The persona ingredient also bears on How do we generate realistic personas at population scale?, which leaves open whether demographic, psychographic or behavioral information is essential. Here behavioral grounding is the design choice, and public data from the same domain does about as well as proprietary data. The target is also a population-level direction, not an individual's response, so it does not conflict with Does conditioning LLMs on personal profiles improve prediction?.

The excerpt leaves a lot open. It gives no baseline for the accuracy figures, no comparison against synthetic or rule-based personas despite the framing, and no numbers for the depth-versus-diversity or subsampling questions. It does not say which model was used, how many personas, or how "large" and "near-zero" effects are defined; the sentence on near-zero effects is cut off mid-thought. The pre-screening uses, filtering "clearly inferior treatment candidates" and ranking proposed changes by predicted impact, are described as potential applications and not tested. The defensible reading is narrow: behavior-grounded personas can flag large, high-salience effects cheaply, and a near-zero predicted effect still needs real traffic to settle.