Shuo Huang
Author directoryOther people with similar names: Shuo Huang
Unverified author pages with similar names: Shuo Huang
2026
One Instruction Is a Benchmark: End-to-End Instruction-Following Evaluation with FlexBench
Yuchen Ge | Yizhao Sun | Wei He | Jinpeng Ou | Wenzhe Niu | Chengshun Shi | Huichuan Fan | Biao Ma | Shuo Huang | Xin Tian | Jiuchong Gao | Jinghua Hao | Renqing He
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Yuchen Ge | Yizhao Sun | Wei He | Jinpeng Ou | Wenzhe Niu | Chengshun Shi | Huichuan Fan | Biao Ma | Shuo Huang | Xin Tian | Jiuchong Gao | Jinghua Hao | Renqing He
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Real-world deployments of large language models (LLMs) increasingly rely on user-provided instructions that are long, precise, and tailored to specific workflows. In such settings, aggregate leaderboard performance is often a weak proxy for a model’s ability to faithfully execute the particular instruction that a practitioner intends to deploy. To address this gap, we introduce FlexBench/FlexEval, an instruction-conditioned evaluation framework that instantiates a specialized test suite from a user-provided target instruction. Given a target instruction, FlexBench (i) derives a set of verifiable evaluation dimensions and (ii) generates an instruction-conditioned conversation corpus. FlexEval then aggregates dimension-level judgments into two complementary metrics: Coverage, which measures workflow progress, and Achievement, which measures conditional correctness. We evaluate the framework on nine multi-turn target instructions spanning diverse scenario categories and workflow topologies, across nine contemporary LLMs over 3,888 generated dialogues. The results show that model rankings and failure modes vary substantially across target instructions, highlighting the need for instruction-conditioned evaluation beyond fixed aggregate benchmarks. Our source code is publicly available at https://github.com/Coryyy/FlexBench.