@inproceedings{shi-etal-2026-sage,
title = "{SAGE}: An Extensible Benchmark for Service Agent Graph-guided Evaluation",
author = "Shi, Ling and
Dai, Yuqin and
Wang, Ziyin and
Zhang, Wei and
Gao, Ning and
Wang, Chaozheng and
Wu, Xinwei and
Wang, Yujie and
He, Wei and
Wang, Jinpeng and
Xu, Linlong and
Xiong, Deyi",
editor = "Lee, Hung-Yi and
Dev, Sunipa and
Li, Jessy and
Elliott, Desmond",
booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {EMNLP} 2026",
month = oct,
year = "2026",
address = "Budapest, Hungary",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.findings-emnlp.1191/",
pages = "25145--25164",
ISBN = "979-8-89176-462-0",
abstract = "LLMs are transforming customer service, yet evaluating service agents across diverse business scenarios remains costly and fragmented, as each new domain requires building a dedicated benchmark from scratch. Existing benchmarks rely on static, scenario-specific scripts with two fundamental limitations: ***Scenario Silos***, where each domain demands costly independent annotation with no shared evaluation structure, and ***Dialogue Dynamics Gap***, where fixed scripts fail to capture the full range of real user behaviors. We propose **SAGE** (**S**ervice **A**gent **G**raph-guided **E**valuation), to our knowledge the first scenario-extensible benchmark to jointly assess logical SOP compliance and conversational quality, abstracting any SOP as a directed graph and bridging Scenario Silos through a lightweight two-step extension mechanism. SAGE further enables dynamic multi-turn simulation via adversarial user agents to overcome the Dynamics Gap. Across six industrial scenarios representative of real-world customer service deployment, we evaluate 27 mainstream LLMs spanning both closed-source systems and open-source families, offering a fine-grained capability map to guide real-world model selection. Our experiments further reveal two systematic failure patterns shared across all tested models: while models generally succeed at recognizing user intent, they frequently struggle to follow complex business logic and execute the correct subsequent action; and they sustain a warm and empathetic conversational tone even when their underlying procedural reasoning has already failed. **Code and resources are available at: https://github.com/lingshi6565/SAGE-Bench.**"
}<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="shi-etal-2026-sage">
<titleInfo>
<title>SAGE: An Extensible Benchmark for Service Agent Graph-guided Evaluation</title>
</titleInfo>
<name type="personal">
<namePart type="given">Ling</namePart>
<namePart type="family">Shi</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Yuqin</namePart>
<namePart type="family">Dai</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Ziyin</namePart>
<namePart type="family">Wang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Wei</namePart>
<namePart type="family">Zhang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Ning</namePart>
<namePart type="family">Gao</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Chaozheng</namePart>
<namePart type="family">Wang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Xinwei</namePart>
<namePart type="family">Wu</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Yujie</namePart>
<namePart type="family">Wang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Wei</namePart>
<namePart type="family">He</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Jinpeng</namePart>
<namePart type="family">Wang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Linlong</namePart>
<namePart type="family">Xu</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Deyi</namePart>
<namePart type="family">Xiong</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<originInfo>
<dateIssued>2026-10</dateIssued>
</originInfo>
<typeOfResource>text</typeOfResource>
<relatedItem type="host">
<titleInfo>
<title>Findings of the Association for Computational Linguistics: EMNLP 2026</title>
</titleInfo>
<name type="personal">
<namePart type="given">Hung-Yi</namePart>
<namePart type="family">Lee</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Sunipa</namePart>
<namePart type="family">Dev</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Jessy</namePart>
<namePart type="family">Li</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Desmond</namePart>
<namePart type="family">Elliott</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<originInfo>
<publisher>Association for Computational Linguistics</publisher>
<place>
<placeTerm type="text">Budapest, Hungary</placeTerm>
</place>
</originInfo>
<genre authority="marcgt">conference publication</genre>
<identifier type="isbn">979-8-89176-462-0</identifier>
</relatedItem>
<abstract>LLMs are transforming customer service, yet evaluating service agents across diverse business scenarios remains costly and fragmented, as each new domain requires building a dedicated benchmark from scratch. Existing benchmarks rely on static, scenario-specific scripts with two fundamental limitations: ***Scenario Silos***, where each domain demands costly independent annotation with no shared evaluation structure, and ***Dialogue Dynamics Gap***, where fixed scripts fail to capture the full range of real user behaviors. We propose **SAGE** (**S**ervice **A**gent **G**raph-guided **E**valuation), to our knowledge the first scenario-extensible benchmark to jointly assess logical SOP compliance and conversational quality, abstracting any SOP as a directed graph and bridging Scenario Silos through a lightweight two-step extension mechanism. SAGE further enables dynamic multi-turn simulation via adversarial user agents to overcome the Dynamics Gap. Across six industrial scenarios representative of real-world customer service deployment, we evaluate 27 mainstream LLMs spanning both closed-source systems and open-source families, offering a fine-grained capability map to guide real-world model selection. Our experiments further reveal two systematic failure patterns shared across all tested models: while models generally succeed at recognizing user intent, they frequently struggle to follow complex business logic and execute the correct subsequent action; and they sustain a warm and empathetic conversational tone even when their underlying procedural reasoning has already failed. **Code and resources are available at: https://github.com/lingshi6565/SAGE-Bench.**</abstract>
<identifier type="citekey">shi-etal-2026-sage</identifier>
<location>
<url>https://aclanthology.org/2026.findings-emnlp.1191/</url>
</location>
<part>
<date>2026-10</date>
<extent unit="page">
<start>25145</start>
<end>25164</end>
</extent>
</part>
</mods>
</modsCollection>
%0 Conference Proceedings
%T SAGE: An Extensible Benchmark for Service Agent Graph-guided Evaluation
%A Shi, Ling
%A Dai, Yuqin
%A Wang, Ziyin
%A Zhang, Wei
%A Gao, Ning
%A Wang, Chaozheng
%A Wu, Xinwei
%A Wang, Yujie
%A He, Wei
%A Wang, Jinpeng
%A Xu, Linlong
%A Xiong, Deyi
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Findings of the Association for Computational Linguistics: EMNLP 2026
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-462-0
%F shi-etal-2026-sage
%X LLMs are transforming customer service, yet evaluating service agents across diverse business scenarios remains costly and fragmented, as each new domain requires building a dedicated benchmark from scratch. Existing benchmarks rely on static, scenario-specific scripts with two fundamental limitations: ***Scenario Silos***, where each domain demands costly independent annotation with no shared evaluation structure, and ***Dialogue Dynamics Gap***, where fixed scripts fail to capture the full range of real user behaviors. We propose **SAGE** (**S**ervice **A**gent **G**raph-guided **E**valuation), to our knowledge the first scenario-extensible benchmark to jointly assess logical SOP compliance and conversational quality, abstracting any SOP as a directed graph and bridging Scenario Silos through a lightweight two-step extension mechanism. SAGE further enables dynamic multi-turn simulation via adversarial user agents to overcome the Dynamics Gap. Across six industrial scenarios representative of real-world customer service deployment, we evaluate 27 mainstream LLMs spanning both closed-source systems and open-source families, offering a fine-grained capability map to guide real-world model selection. Our experiments further reveal two systematic failure patterns shared across all tested models: while models generally succeed at recognizing user intent, they frequently struggle to follow complex business logic and execute the correct subsequent action; and they sustain a warm and empathetic conversational tone even when their underlying procedural reasoning has already failed. **Code and resources are available at: https://github.com/lingshi6565/SAGE-Bench.**
%U https://aclanthology.org/2026.findings-emnlp.1191/
%P 25145-25164
Markdown (Informal)
[SAGE: An Extensible Benchmark for Service Agent Graph-guided Evaluation](https://aclanthology.org/2026.findings-emnlp.1191/) (Shi et al., Findings 2026)
ACL
- Ling Shi, Yuqin Dai, Ziyin Wang, Wei Zhang, Ning Gao, Chaozheng Wang, Xinwei Wu, Yujie Wang, Wei He, Jinpeng Wang, Linlong Xu, and Deyi Xiong. 2026. SAGE: An Extensible Benchmark for Service Agent Graph-guided Evaluation. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 25145–25164, Budapest, Hungary. Association for Computational Linguistics.