Huawei Feng

Author directory

2026

While recent medical benchmarks for Large Language Models (LLMs) have begun to incorporate clinical temporality, they still overlook a key reality: clinical evidence is encountered incrementally, requiring repeated decisions between reviewing the next available record and committing to a prediction. To address this issue, we introduce [<https://github.com/hwfengcs/INA-Bench>] , a benchmark for Inpatient Note Acquisition that evaluates LLMs’ ability to make acquire-or-predict decisions using sequentially revealed real-world Chinese inpatient records. In this benchmark, each case sequentially reveals up to three note blocks: an admission note, a first-course summary, and a daily-progress summary. After each block, the model under evaluation must either request the next block or predict one or more later documented follow-up actions (laboratory, imaging, pathology, or consultation). Surgery and discharge records are withheld for label construction and boundary checks only. is built from 8,986 de-identified inpatient records collected from two tertiary teaching hospitals, with patient-disjoint and temporal evaluation splits. Representative LLMs exhibit diverse acquire-or-predict behaviors. For example, DeepSeek-chat improves action micro-F1 by +0.102 and +0.074 on the patient-disjoint and temporal splits, respectively, while reading fewer note blocks, though the gains weaken under temporal drift.
Reliable clinical decision support requires tracking evidence over a hospital stay and revising a disease diagnosis when new findings challenge the initial assessment. Strong performance on medical question answering does not establish that large language models (LLMs) can make these longitudinal decisions. We introduce DiagBench, a benchmark of 2,865 Chinese inpatient admissions that evaluates diagnostic revision as clinical evidence accumulates. It pairs admission-only prediction with staged evidence disclosure, distinguishing an unchanged diagnosis, refinement of an existing hypothesis, and replacement by a different diagnosis. Development and test cases have human-annotated reference labels. The best LLM configuration in the main evaluation achieves only 52.6% macro recall; 80.3-87.7% of errors across five DeepSeek-Chat (DeepSeek-V3.2, non-thinking mode) prompting strategies occur between adjacent revision categories. Strategies that improve final-diagnosis prediction do not consistently improve revision decisions. We further use DiagAgent to trace diagnoses and evidence across rounds, revealing failures to replace an initial hypothesis despite recognizing contradictory findings. On DeepSeek-Chat, a single agent with the same state outperforms the full system on macro recall.
Search