Yang Pengtao
Author directory2026
RealFin: Benchmarking Agent Workflow Orchestration Beyond MCP-Style Tool Connectivity
Ying Huang | Shuoling Liu | Liyuan Chen | Jie Shi | Jiaqing Liang | Yang Pengtao | Ying Liao | Jinyi Han | Sihang Jiang | Yanghua Xiao
Findings of the Association for Computational Linguistics: EMNLP 2026
Ying Huang | Shuoling Liu | Liyuan Chen | Jie Shi | Jiaqing Liang | Yang Pengtao | Ying Liao | Jinyi Han | Sihang Jiang | Yanghua Xiao
Findings of the Association for Computational Linguistics: EMNLP 2026
Standardized interfaces such as the Model Context Protocol (MCP) make it easier to connect tools to agents, but connection alone is not orchestration. When many related tools are exposed as a flat callable palette, agents must identify relevant tools, resolve substitutes, pass intermediate outputs, and decide when a workflow is complete. We introduce RealFin, a finance-domain diagnostic benchmark for evaluating workflow construction under MCP-style flat tool exposure. RealFin contains 1524 tasks generated from 108 author-designed financial workflow templates over real market, macroeconomic, and company-data APIs. Each task is paired with an executable reference program that specifies one verified solution path, while the 85 agent-callable tools include explicit alternative, dependency, and related-tool metadata to represent non-unique paths, prerequisite tools, and near-neighbor ambiguity. This design supports path-independent answer scoring together with process-level evaluation of tool selection, ordering, parameter planning, entity grounding, temporal grounding, orchestration success, and error recovery. Across ReAct-style flat exposure, retrieval-based exposure, oracle-controlled variants, and progressive tool disclosure, current agents often achieve nonzero answer accuracy while failing to construct reliable workflows. These results suggest that MCP-style tool connectivity needs structured disclosure and workflow-aware evaluation, especially in finance where temporal constraints and overlapping tools are intrinsic.