Zhijie Bao
Author directory2026
ObGynLongBench: Revealing the Evidence-to-EHR Gap in Longitudinal EHR Decision-Making
Jun Xiang | Zhijie Bao | Hu Rong | Kaizhou Qin | Wei Chen | Zhongyu Wei
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Jun Xiang | Zhijie Bao | Hu Rong | Kaizhou Qin | Wei Chen | Zhongyu Wei
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
The application of large language models (LLMs) to personalized medical assistants has garnered growing interest. However, existing medical benchmarks largely rely on static question answering with pre-selected evidence, leaving unclear whether LLMs can make reliable clinical decisions from real longitudinal electronic health records (EHRs). To bridge this gap, we introduce ObGynLongBench, a rule-grounded long-context EHR benchmark for obstetric and gynecologic decision-making, comprising 1,500 clinical decision-point cases from 976 real pregnancy EHR histories and traceable rules. Each case is anchored to a patient, a pregnancy-timeline point, and a pre-decision information boundary, enabling Evidence-only, Visit-level EHR, and History-level EHR evaluation. Evaluating 17 LLMs reveals a substantial Evidence-to-EHR Gap: models perform well when evidence is directly provided, but accuracy drops when evidence must be extracted from same-day records or full pre-decision EHR histories. Further analyses identify evidence utilization as a key bottleneck: performance decreases with longer EHR contexts and more complex evidence requirements, and earlier failures often predict later failures within the same patient history. Finally, active-search agents perform best among EHR access strategies, highlighting patient-specific evidence utilization as a central challenge for reliable personalized medical assistants.
2025
SFMSS: Service Flow aware Medical Scenario Simulation for Conversational Data Generation
Zhijie Bao | Qingyun Liu | Xuanjing Huang | Zhongyu Wei
Findings of the Association for Computational Linguistics: NAACL 2025
Zhijie Bao | Qingyun Liu | Xuanjing Huang | Zhongyu Wei
Findings of the Association for Computational Linguistics: NAACL 2025
Medical-specific Large Language Models (LLMs) have demonstrated impressive performance on medical-related exams and tasks. Despite their success in single-turn question and answering, instruction-tuned LLMs often falter in real-world healthcare applications, highlighting a disconnect between existing instruction datasets and practical contexts. To address this issue, we propose Service Flow aware Medical Scenario Simulation (SFMSS), a simulation framework designed for medical conversational data generation. SFMSS employs three key strategies to ensure the quality of the data generation. the use of Authentic Seed Data ensures alignment of real-world distributions. Diverse Patient Simulation enables simulated patients to exhibit distinct communication styles and complex behavioral logic. Service Flow Control ensures that conversations progress in alignment with medical objectives. We construct a dataset targeting on outpatient reception through SFMSS, named SFMSS-CD. Building on this dataset, we develop a model called SFMSS-Nurse. We conduct both automatic and human evaluations, involving 15 users and 15 clinical experts, to assess the effectiveness of SFMSS. The results demonstrate that SFMSS-Nurse outperforms all baselines, including the current state-of-the-art model GPT-4o, and aligns with human preferences and clinical demands.