Evaluating conversational agents typically requires human participants, a process that is costly and difficult to scale. LLM-based user simulators and LLM-as-a-judge evaluators offer a promising alternative, but how close they are to replacing real users remains unclear. We study this through a human user study (52 participants) and a simulated user study (four LLM-based simulators) applied to an experimental conversational sales agent, comparing LLM-as-a-judge assessments against human self-reported ratings. The results show that none of our simulators fully replicates human interaction patterns, and none of the LLM-based evaluators shows strong correlation with human ratings on real dialogues. We provide practical guidelines for combining automated and human evaluation, identifying key areas for improvement and highlighting when real-user studies remain essential.

We present a novel approach to conversational agent evaluation using Persona-driven User Simulations based on Large Language Models (LLMs). Our methodology first uses LLMs to generate diverse customer personas, which are then used to configure a single LLM-based user simulator. This simulator evaluates SalesBot 2.0, a proactive conversational sales agent. We introduce a dataset of these personas, along with corresponding goals and conversation scenarios, enabling comprehensive testing across different customer types with varying assertiveness levels and precision of needs. Our evaluation framework assesses both the simulator’s adherence to persona instructions and the bot’s performance across multiple dimensions, combining human annotation with LLM-as-a-judge assessments using commercial and open-source models. Results demonstrate that our LLM-based simulator effectively emulates nuanced customer roles, and that cross-selling strategies can be implemented with minimal impact on customer satisfaction, varying by customer type.