Ziyin Wang
Author directory2026
SAGE: An Extensible Benchmark for Service Agent Graph-guided Evaluation
Ling Shi | Yuqin Dai | Ziyin Wang | Wei Zhang | Ning Gao | Chaozheng Wang | Xinwei Wu | Yujie Wang | Wei He | Jinpeng Wang | Linlong Xu | Deyi Xiong
Findings of the Association for Computational Linguistics: EMNLP 2026
Ling Shi | Yuqin Dai | Ziyin Wang | Wei Zhang | Ning Gao | Chaozheng Wang | Xinwei Wu | Yujie Wang | Wei He | Jinpeng Wang | Linlong Xu | Deyi Xiong
Findings of the Association for Computational Linguistics: EMNLP 2026
LLMs are transforming customer service, yet evaluating service agents across diverse business scenarios remains costly and fragmented, as each new domain requires building a dedicated benchmark from scratch. Existing benchmarks rely on static, scenario-specific scripts with two fundamental limitations: ***Scenario Silos***, where each domain demands costly independent annotation with no shared evaluation structure, and ***Dialogue Dynamics Gap***, where fixed scripts fail to capture the full range of real user behaviors. We propose **SAGE** (**S**ervice **A**gent **G**raph-guided **E**valuation), to our knowledge the first scenario-extensible benchmark to jointly assess logical SOP compliance and conversational quality, abstracting any SOP as a directed graph and bridging Scenario Silos through a lightweight two-step extension mechanism. SAGE further enables dynamic multi-turn simulation via adversarial user agents to overcome the Dynamics Gap. Across six industrial scenarios representative of real-world customer service deployment, we evaluate 27 mainstream LLMs spanning both closed-source systems and open-source families, offering a fine-grained capability map to guide real-world model selection. Our experiments further reveal two systematic failure patterns shared across all tested models: while models generally succeed at recognizing user intent, they frequently struggle to follow complex business logic and execute the correct subsequent action; and they sustain a warm and empathetic conversational tone even when their underlying procedural reasoning has already failed. **Code and resources are available at: https://github.com/lingshi6565/SAGE-Bench.**