LLM-based voice agents are routinely cleared for deployment using synthetic audio regression suites, with engineering teams treating augmented commercial TTS as a safe proxy for real human callers. This paper presents the first systematic approach to quantify the profound behavioral drift that occurs when agents optimized on simulated audio are deployed in the real world. We demonstrate that the assumption of acoustic equivalence fundamentally fails: real production audio triggers 2-13 times more VAD misfires than synthetic data across Silero, WebRTC, and TEN-VAD, while simulated environments cause text-grounded turn detectors to artificially flip decisions 33% of the time, compared to just 3% in reality. A paired four-vendor TTS calibration sweep proves this misranking is structural: on every front-end component, optimizing thresholds for synthetic audio actively degrades real-world performance. To prevent teams from flying blind, we introduce a novel set of signal-layer metrics and an open-source framework to audit any simulator prior to adoption. Rather than testing the agent, our method tests the simulator against real target-domain interactions. By identifying where a simulator’s acoustics deviate from reality, teams can definitively know which front-end tuning decisions are safe to trust.

Entity recognition in Automatic Speech Recognition (ASR) is challenging for rare and domain-specific terms. In domains such as finance, medicine, and air-traffic control, these errors are costly. If an entity is entirely absent from the ASR output, text-only post-correction has little signal to recover it. To address this, we introduce RECOVER, an agentic post-ASR entity-correction framework. It leverages multiple hypotheses as evidence from ASR, retrieves relevant entities, proposes Large Language Model (LLM) correction under constraints, filters them through deterministic guardrails and uses an LLM critic that judges each surviving edit and can trigger a targeted feedback round to recover an entity the first pass missed. The hypotheses are used by one of several strategies, namely 1-Best, Entity-Aware Select, Recognizer Output Voting Error Reduction (ROVER) Ensemble, LLM-Select and Adaptive Routing. Evaluated across five diverse datasets over a Whisper-large-v3 baseline, it achieves up to 49% relative reduction in entity-phrase word error rate (E-WER) and increases entity recall by up to 14 percentage points while preserving the overall word error rate (WER), with LLM-Select the strongest strategy.

Traditional customer support systems, such as Interactive Voice Response (IVR), rely on rigid scripts and lack the flexibility required for handling complex, policy-driven tasks. While large language model (LLM) agents offer a promising alternative, evaluating their ability to act in accordance with business rules and real-world support workflows remains an open challenge. Existing benchmarks primarily focus on tool usage or task completion, overlooking an agent’s capacity to adhere to multi-step policies, navigate task dependencies, and remain robust to unpredictable user or environment behavior. In this work, we introduce JourneyBench, a benchmark designed to assess policy-aware agents in customer support. JourneyBench leverages graph representations to generate diverse, realistic support scenarios and proposes the User Journey Coverage Score, a novel metric to measure policy adherence. We evaluate multiple state-of-the-art LLMs using two agent designs: a Static-Prompt Agent (SPA) and a Dynamic-Prompt Agent (DPA) that explicitly models policy control. Across 703 conversations in three domains, we show that DPA significantly boosts policy adherence, even allowing smaller models like GPT-4o-mini to outperform more capable ones like GPT-4o. Our findings demonstrate the importance of structured orchestration and establish JourneyBench as a critical resource to advance AI-driven customer support beyond IVR-era limitations.

Recent advancements in large Language Models (LMs) have significantly enhanced their capabilities across various domains, including natural language understanding and generation. In this paper, we investigate the application of LMs to the specialized task of contact-center Quality Assurance (QA), which involves evaluating conversations between human agents and customers. This task requires both sophisticated linguistic understanding and deep domain knowledge. We conduct a comprehensive assessment of eight LMs, revealing that larger models, such as Claude-3.5-Sonnet, exhibit superior performance in comprehending contact-center conversations. We introduce methodologies to transfer this domain-specific knowledge to smaller models by leveraging evaluation plans generated by more knowledgeable models, with optional human-in-the-loop refinement to enhance the capabilities of smaller models. Notably, our experimental results demonstrate an improvement of up to 18.95% in Macro F1 on an in-house QA dataset. Our findings emphasize the importance of evaluation plans in guiding reasoning and highlight the potential of AI-assisted tools to advance objective, consistent, and scalable agent evaluation processes in contact centers.