Hefei Chen
Author directory2026
Meeseeks-Multilingual: A Multi-Turn, Multi-Constraint Benchmark evaluating Multilingual Instruction-Following and Self-Correction
Jiaming Wang | Hefei Chen | Zhe Tang | Zehao Jin | Yanxin Zhuo | Ruicheng Liao | Jinrong Ma | Xiaoyu Li | Xuezhi Cao
Findings of the Association for Computational Linguistics: EMNLP 2026
Jiaming Wang | Hefei Chen | Zhe Tang | Zehao Jin | Yanxin Zhuo | Ruicheng Liao | Jinrong Ma | Xiaoyu Li | Xuezhi Cao
Findings of the Association for Computational Linguistics: EMNLP 2026
Large Language Models (LLMs) fail to reliably follow multi-constraint instructions across languages, and the failure rate rises when regional linguistic rules interact with general constraints. To address this challenge, we present Meeseeks-Multilingual, a multi-turn, feedback-driven benchmark that extends Meeseeks to multilingual and cross-regional evaluation. The benchmark separates general constraints (e.g., word counts, keyword rules) from regional constraints (e.g., Japanese kana requirements) so that the two failure modes can be measured independently. Meeseeks-Multilingual automatically identifies unmet constraints and provides targeted feedback to test iterative self-correction. All data are produced by trained annotators with no LLM involvement in data generation. Experiments across mainstream commercial and open-source LLMs show that even with iterative feedback, models consistently fail to meet all multilingual constraints. Aggregate model rankings stabilize by Round 5, which lets the benchmark be run at reduced cost for overall comparisons, although regional rankings settle more slowly and need more rounds. Meeseeks-Multilingual has been open-sourced on https://github.com/meituan-longcat/Meeseeks.
SOP-Maze: Evaluating Large Language Models on Complicated Business Standard Operating Procedures
Jiaming Wang | Zhe Tang | Zehao Jin | Hefei Chen | Yilin Jin | Peng Ding | Xiaoyu Li | Xuezhi Cao
Findings of the Association for Computational Linguistics: ACL 2026
Jiaming Wang | Zhe Tang | Zehao Jin | Hefei Chen | Yilin Jin | Peng Ding | Xiaoyu Li | Xuezhi Cao
Findings of the Association for Computational Linguistics: ACL 2026
As large language models (LLMs) are widely deployed as domain-specific agents, many benchmarks have been proposed to evaluate their ability to follow instructions and make decisions in real-world scenarios. However, business scenarios often involve complex standard operating procedures (SOPs), and the evaluation of LLM capabilities in such contexts has not been fully explored. To bridge this gap, we propose SOP-Maze, a benchmark constructed from real-world business data and adapted into a collection of 397 instances and 3422 subtasks from 23 complex SOP scenarios. We further categorize SOP tasks into two broad classes: Lateral Root System (LRS), representing wide-option tasks that demand precise selection; and Heart Root System (HRS), which emphasizes deep logical reasoning with complex branches. Extensive experiments reveal that nearly all state-of-the-art models struggle with SOP-Maze. We conduct a comprehensive analysis and identify three key error categories: (i) route blindness: difficulty following procedures; (ii) conversational fragility: inability to handle real dialogue nuances; and (iii) calculation errors: mistakes in time or arithmetic reasoning under complex contexts. The systematic study explores LLM performance across SOP tasks that challenge both breadth and depth, offering new insights for improving model capabilities. We have open-sourced our work on the anonymous link: https://github.com/meituan-longcat/SOP-Maze.