Ziyad Benomar
Author directory2026
Data-Driven Persona-Conditioned Agents for A/B Test Simulation
Ziyad Benomar | Weronika Łajewska | Leonardo Perelli | Saab Mansour
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
Ziyad Benomar | Weronika Łajewska | Leonardo Perelli | Saab Mansour
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
A/B testing is the gold standard for evaluating product changes, but each experiment requires real user traffic, engineering effort, and weeks of measurement. We propose a simulation framework that predicts A/B test outcomes using LLM-powered agents conditioned on data-driven personas grounded in real user behavioral signals. Unlike prior work that relies on synthetic or rule-based personas, our agents are constructed from anonymized behavioral data—activity patterns, engagement signals, and inferred demographics—enabling more faithful population modeling. We frame A/B test simulation as a structured question task and systematically study (i) question design formats, (ii) the impact of persona data source and domain alignment, (iii) the trade-off between per-persona behavioral depth and population diversity, and (iv) efficient population subsampling. On a benchmark of 40 A/B tests spanning two metric types, our best configuration achieves 0.75–0.90 directional accuracy depending on the test metric, demonstrating that data-driven personas are a viable path toward fast, low-cost experiment pre-screening.