Understanding drivers of customer satisfaction in sales conversations is crucial for dialogue system optimization and customer experience improvement, yet existing neural approaches lack interpretability. We introduce a sales action annotation framework that tracks offers, cross-sells, and customer decisions in sales dialogues. From these structured annotations, we derive interpretable metrics spanning conversion rates, timing and negotiation patterns, and sales outcomes to predict user satisfaction. We validate our metrics-based regression model against a transformer-based text-embedding baseline on 5200 simulated phone sales dialogues. Our interpretable model achieves strong performance (R²=0.87 vs. 0.92 for the baseline) with full transparency, identifying key satisfaction drivers with practical implications for dialogue design: purchase completion, conversation efficiency, and negotiation patterns.

Evaluating conversational agents typically requires human participants, a process that is costly and difficult to scale. LLM-based user simulators and LLM-as-a-judge evaluators offer a promising alternative, but how close they are to replacing real users remains unclear. We study this through a human user study (52 participants) and a simulated user study (four LLM-based simulators) applied to an experimental conversational sales agent, comparing LLM-as-a-judge assessments against human self-reported ratings. The results show that none of our simulators fully replicates human interaction patterns, and none of the LLM-based evaluators shows strong correlation with human ratings on real dialogues. We provide practical guidelines for combining automated and human evaluation, identifying key areas for improvement and highlighting when real-user studies remain essential.

We present a novel approach to conversational agent evaluation using Persona-driven User Simulations based on Large Language Models (LLMs). Our methodology first uses LLMs to generate diverse customer personas, which are then used to configure a single LLM-based user simulator. This simulator evaluates SalesBot 2.0, a proactive conversational sales agent. We introduce a dataset of these personas, along with corresponding goals and conversation scenarios, enabling comprehensive testing across different customer types with varying assertiveness levels and precision of needs. Our evaluation framework assesses both the simulator’s adherence to persona instructions and the bot’s performance across multiple dimensions, combining human annotation with LLM-as-a-judge assessments using commercial and open-source models. Results demonstrate that our LLM-based simulator effectively emulates nuanced customer roles, and that cross-selling strategies can be implemented with minimal impact on customer satisfaction, varying by customer type.