Huichuan Fan

Author directory

2026

Real-world deployments of large language models (LLMs) increasingly rely on user-provided instructions that are long, precise, and tailored to specific workflows. In such settings, aggregate leaderboard performance is often a weak proxy for a model’s ability to faithfully execute the particular instruction that a practitioner intends to deploy. To address this gap, we introduce FlexBench/FlexEval, an instruction-conditioned evaluation framework that instantiates a specialized test suite from a user-provided target instruction. Given a target instruction, FlexBench (i) derives a set of verifiable evaluation dimensions and (ii) generates an instruction-conditioned conversation corpus. FlexEval then aggregates dimension-level judgments into two complementary metrics: Coverage, which measures workflow progress, and Achievement, which measures conditional correctness. We evaluate the framework on nine multi-turn target instructions spanning diverse scenario categories and workflow topologies, across nine contemporary LLMs over 3,888 generated dialogues. The results show that model rankings and failure modes vary substantially across target instructions, highlighting the need for instruction-conditioned evaluation beyond fixed aggregate benchmarks. Our source code is publicly available at https://github.com/Coryyy/FlexBench.