Kailuan Wu
Author directory2026
MedCPEnv: A Reproducible Cost-Aware Simulated Hospital for Benchmarking and Training Medical LLM Agents
Zixiao Jia | Changjie Hua | Weiqi Zhai | Ruyi Xu | Xue Yang | Han Li | Tianyun Lu | Kailuan Wu | Xiang Zheng | Lin Qu | hu Wei | Bing Zhao | Ruohan Zhang
Findings of the Association for Computational Linguistics: EMNLP 2026
Zixiao Jia | Changjie Hua | Weiqi Zhai | Ruyi Xu | Xue Yang | Han Li | Tianyun Lu | Kailuan Wu | Xiang Zheng | Lin Qu | hu Wei | Bing Zhao | Ruohan Zhang
Findings of the Association for Computational Linguistics: EMNLP 2026
Real-world clinical competence demands more than answering medical knowledge questions—it requires proactively gathering incomplete patient information, selecting appropriate examinations, and formulating safe, cost-conscious treatment plans through multi-turn interaction. Existing medical LLM benchmarks rely predominantly on static QA formats, failing to assess interactive information acquisition, tool use, or cost–quality tradeoffs. We propose MedCPEnv, a reproducible, cost-aware simulated hospital environment for evaluating and training medical LLM agents. Built from MIMIC-IV-NOTE discharge notes, MedCPEnv comprises 48,803 curated cases, a 1,000-case benchmark, and a 371,550-entry medical QA knowledge base, exposing four deterministic action types (ASK, EXAM, KNOWLEDGE, FINAL) to eliminate high-variance LLM role-playing artifacts. Evaluating eight frontier models, GPT-5.4 achieves the highest composite score (3.69/5) and diagnostic accuracy (60.3%), yet treatment accuracy remains only 19.4%, identifying treatment plan generation as the primary bottleneck. Substantial inter-model variation in the cost–quality tradeoff further demonstrates that clinical correctness alone is insufficient to characterize agent behavior. GRPO-based Agentic RL on Qwen3-4B shows that medical quality rewards consistently improve tool-use strategies, raising the composite score from 2.97 to 3.14 and diagnostic accuracy from 26.0% to 31.2%, establishing MedCPEnv as both a rigorous evaluation benchmark and a reproducible training environment for medical agent research.
ClinConsensus: A Physician-Calibrated Benchmark for Evaluating Clinical Rubric Coverage in Chinese Medical LLMs
Xiang Zheng | Han Li | Wenjie Luo | Weiqi Zhai | Yiyuan Li | Chuanmiao Yan | Xue Yang | Kailuan Wu | Ruiyi Xu | Tianyun Lu | Tianyi Tang | Yubo Ma | Kexin Yang | Sen Yang | Dayiheng Liu | Lin Qu | Bing Zhao | HU Wei
Findings of the Association for Computational Linguistics: EMNLP 2026
Xiang Zheng | Han Li | Wenjie Luo | Weiqi Zhai | Yiyuan Li | Chuanmiao Yan | Xue Yang | Kailuan Wu | Ruiyi Xu | Tianyun Lu | Tianyi Tang | Yubo Ma | Kexin Yang | Sen Yang | Dayiheng Liu | Lin Qu | Bing Zhao | HU Wei
Findings of the Association for Computational Linguistics: EMNLP 2026
Open-ended medical LLM evaluation remains weakly grounded in physician-calibrated coverage of clinically relevant response criteria, especially in localized clinical settings. We introduce ClinConsensus, a Chinese medical benchmark of 2,500 expert-curated cases spanning 36 specialties, 12 task themes, multiple difficulty levels, and lay-facing versus professional-facing settings. Each case is paired with 30 case-specific binary rubric criteria. To evaluate whether responses satisfy enough physician-authored criteria, we propose Clinician-Anchored Coverage Score (CACS), a physician-calibrated threshold metric instantiated at k = 10, and develop a dual-judge framework combining a GPT-5.1 grader with a physician-supervised Qwen3-8B judge. Evaluating 11 frontier LLMs, we find a persistent coverage gap: Rubric Accuracy ranges from 39.6% to 52.1%, whereas CACS@10 ranges from 17.8% to 32.9%, leaving a 19.2–21.9 point gap across models. Stratified analyses further reveal substantial variation across reasoning, evidence use, structured extraction, medication instructions, follow-up, and dialogue register. These results suggest that medical LLM evaluation should measure thresholded, rubric-grounded clinical coverage rather than average partial correctness.