Xing Fan
Author directoryOther people with similar names: Xing Fan
Unverified author pages with similar names: Xing Fan
2026
Beyond Perfect APIs: Evaluating LLM Agents Under Realistic API Complexity
Doyoung Kim | Zhiwei Ren | Jie Hao | Zhongkai Sun | Lichao Wang | Xiyao Ma | Xu Han | Jun Yin | Heng Ji | Wei Shen | Xing Fan | Benjamin Z. Yao | Chenlei Guo
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Doyoung Kim | Zhiwei Ren | Jie Hao | Zhongkai Sun | Lichao Wang | Xiyao Ma | Xu Han | Jun Yin | Heng Ji | Wei Shen | Xing Fan | Benjamin Z. Yao | Chenlei Guo
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
The rise of fully autonomous agents necessitates a rigorous evaluation of their reliability under **real-world API complexity**. Accordingly, moving beyond perfectly **sanitized** settings that disregard realistic obstacles, we introduce ***WildAGTEval***, a configurable benchmark environment that accounts for two critical dimensions of realistic API complexity: **API specification**, which includes detailed documentation and usage constraints, and **API execution**, which captures runtime challenges. Consequently, ***WildAGTEval*** offers (i) an API system spanning 86 functions across 7 domains with 60 independently activatable complexity scenarios, yielding approximately 32K distinct test configurations, and (ii) 300 multi-turn conversations grounded in this system. Through systematic evaluation of 11 state-of-the-art LLMs, we find that: (1) individual complexities reduce accuracy by up to 23.1%; (2) compounding them amplifies this drop to 52.1%; and crucially, (3) failing agents fabricate success and distort intent rather than admitting limitations, exposing a critical reliability gap invisible to existing benchmarks.
Analyzing and Internalizing Complex Policy Documents for LLM Agents
Jiateng Liu | Zhenhailong Wang | Xiaojiang Huang | Yingjie Li | Xiang Li | Chenlei Guo | Xing Fan | Ruhi Sarikaya | Heng Ji
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Jiateng Liu | Zhenhailong Wang | Xiaojiang Huang | Yingjie Li | Xiang Li | Chenlei Guo | Xing Fan | Ruhi Sarikaya | Heng Ji
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Large language model agents rely on in-context policy documents encoding diverse business rules. As businesses scale, these documents grow, creating substantial computational overhead and motivating internalization methods that embed policy into model priors. Prior work focuses on generic prompts, but we find agentic policies span multiple complexity levels and demand heavier reasoning, posing greater challenges. We introduce an agentic benchmark generator with Controllable Complexity in agent policy across four levels, enabling systematic evaluation of agents under increasing complexity and providing a testbed for policy internalization. Our analysis shows that workflow-governing policy specifications are the hardest to reason over, and that SFT on gold trajectories with chain-of-thought is data-hungry and struggles at high complexity. We propose Category-Aware Policy Continued Pretraining, an automated pipeline that analyzes policies, extracts key specifications, categorizes them into factual, behavioral, and conditional types, and isolates those driving workflow complexity. This enables targeted “therapy” by synthesizing specialized training data for each type and improving internalization via an autoregressive pretraining loss. Extensive experiments show our synthetic data and objective consistently improve performance. Combined with SFT, our method outperforms the baseline across different settings, especially in data-sparse and high-complexity regimes, with gains up to 41% and 22% on Qwen-3-32B. Overall, we achieve 97.3% prompt reduction on our benchmark, and on 𝜏-Bench we further improve performance while reducing prompt requirements with very limited SFT data.