Hari Subramani

Author directory

2026

Voice agents are increasingly deployed across enterprise applications. However, no existing benchmark jointly addresses realistic conversation simulation and comprehensive voice-specific evaluation. We present EVA-Bench, an end-to-end evaluation framework that addresses both. On the simulation side, EVA-Bench orchestrates dynamic bot-to-bot audio conversations with automatic simulation validation that detects user simulator error and appropriately regenerates conversations before scoring. On the measurement side, EVA-Bench introduces two composite metrics: EVA-A (Accuracy) and EVA-X (Experience). EVA-Bench includes 213 scenarios across three enterprise domains, a controlled perturbation suite for accent and noise robustness, and multi-trial measurements that distinguish peak from reliable capability. Across 12 systems spanning three architectures, we find: (1) no system simultaneously exceeds 0.5 on both EVA-A pass@1 and EVA-X pass@1; (2) peak and reliable performance diverge substantially (median pass@k - pass^k gap of 0.44 on EVA-A; and (3) accent and noise perturbations expose substantial robustness gaps, with effects varying across architectures, systems, and metrics (mean Δ up to 0.314). We release EVA-Bench under an open-source license.
While individual components of agentic architectures have been studied in isolation, we lack empirical evidence on how design dimensions interact in realistic multi-agent systems. This study aims to address these gaps by providing a comprehensive enterprise-specific benchmark. We introduce AgentArch: a benchmark that evaluates 18 agentic configurations across state-of-the-art large language models. The study spans four dimensions: orchestration strategy, agent style (ReAct versus function calling), memory architecture, and thinking tool integration. Results reveal strong model-specific architectural preferences that challenge one-size-fits-all assumptions in agentic AI. They also expose notable performance gaps on enterprise tasks: the best models reach only 35.3% success on the complex task and 70.8% on the simpler task. We aim to inform architecting choices for enterprise agents by grounding model selection and component design in systematic evidence.