Large language models (LLMs) have been widely adopted in healthcare, yet they still encounter significant challenges in complex clinical decision-making scenarios. Existing benchmarks primarily assess LLM performance in single-course settings and lack systematic evaluation in multi-course scenarios, where a patient’s condition evolves over time. To address this gap, we propose ClinicalMC, a benchmark for multi-course clinical decision-making. It includes 1,275 Chinese and 5,804 English samples across four stages from admission to discharge. These stages cover triage, first-course examination/diagnosis/treatment, subsequent multi-course examination/assessment/treatment, and final diagnosis. In ClinicalMC, patients in the English dataset undergo an average of 5.11 clinical courses, whereas those in the Chinese dataset undergo 3.42. To assess LLM performance, we construct a multi-agent evaluation framework that includes patient, examiner, and doctor agents. Based on the benchmark and framework, we design two experimental settings—a single-turn static setting and a multi-turn dynamic setting—and assess three categories of LLMs: 1) closed-source LLMs like GPT-4o-mini; 2) open-source LLMs like DeepSeek-V3, and 3) medical LLMs like HuatuoGPT-o1. Through extensive evaluation, we aim to better understand LLM performance in the medical domain and support its effective deployment in healthcare.

Medical insurance plays a pivotal role in the healthcare system, where effective supervision relies on the precise execution of complex medical insurance rules. Existing solutions for medical insurance violation detection mainly rely on costly manual labor, limiting efficiency. Although current Large Language Models (LLMs) have demonstrated significant potential in general Natural Language Processing (NLP) tasks, they still struggle to understand these rules, and a corresponding evaluation benchmark is lacking. Hence, we propose T2ER, a benchmark to evaluate the capability of LLMs in converting texts into executable rules in medical insurance. T2ER consists of 2,115 rules and 5,636 synthetic Electronic Medical Record (EMR) test cases for execution verification. Furthermore, we design a dual evaluation framework spanning rule and execution levels to comprehensively assess transformation quality. We conduct extensive experiments on advanced LLMs. The results reveal a significant human-AI performance gap, indicating that this conversion remains a formidable challenge. The dataset is available at https://github.com/YHJ1223/T2ER.