Shangda Yang
Author directory2026
MACRO: A Benchmark for Clarification-Aware Multi-Agent Orchestration
Tao Li | Jianyi Chen | Yaqiang Wu | Jie Ma | Shangda Yang | Oguz H Elibol | Pradyumna S. Singh
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Tao Li | Jianyi Chen | Yaqiang Wu | Jie Ma | Shangda Yang | Oguz H Elibol | Pradyumna S. Singh
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Clarification-aware multi-agent orchestration is challenging because systems must elicit missing information, maintain task state, and coordinate dependent subtasks. Existing benchmarks do not comprehensively evaluate such orchestration under underspecified user requests, focusing instead on end-to-end success or isolated abilities like tool use and dialogue. We introduce MACRO, a travel-domain benchmark designed to address these gaps by providing structured task tiers, process-level user-side and system-side milestone annotations that include expected user clarifications, and standardized sub-agent interfaces. MACRO-Full contains 107 fully annotated tasks with 703 milestones, and MACRO-Core provides a tractable 30-task subset with 164 milestones. We evaluate three representative orchestration frameworks and a reference decoupled planning–execution orchestrator (MACRO-DPE) using trace-level metrics and failure analysis. Results show that MACRO-DPE improves routing fidelity and execution efficiency while maintaining task completion, whereas staged planning ensures milestone coverage at higher cost. The dataset[<https://huggingface.co/datasets/weier248/MACRO-bench>] and code are publicly available at https://github.com/liuweie/MACRO-bench/.