Libiao Chen
Author directory2026
H2-MRG: Hierarchical Heterogeneous Graph Based Medical Report Generation with Fine-Grained Disease-Specific Modeling
Libiao Chen | Dong Nie | Zhenyu Tang
Findings of the Association for Computational Linguistics: EMNLP 2026
Libiao Chen | Dong Nie | Zhenyu Tang
Findings of the Association for Computational Linguistics: EMNLP 2026
Vision-language model (VLM)-based medical report generation (MRG) has shown great potential for automating radiology workflows. However, generating clinically reliable reports requires each clinical statement to be grounded in anatomically localized visual evidence and supported by structured disease reasoning. Most existing VLMs partition images into coarse and fixed-size patches regardless of anatomical boundaries, producing semantically fragmented visual tokens. Moreover, CLIP-style contrastive pretraining treats all unpaired image-report pairs as negatives, even though images from different patients often describe the same diseases or findings, thereby introducing false negatives and corrupting visual-text alignment. To address these issues, we propose H2-MRG, a hierarchical heterogeneous graph-based medical report generation framework. Specifically, fine-grained image patches containing complete anatomical structures are extracted. Based on the patches, a hierarchical heterogeneous graph attention network (H2-GAT) module is presented, where anatomy and disease nodes are connected and aggregate information through anatomy-anatomy (spatial), anatomy-disease (diagnostic), and disease-disease (co-occurrence) relations, enabling structured disease reasoning where each disease prediction is grounded in anatomically localized visual evidence. In addition, an anatomy-level sampling strategy for patch-sentence-level alignment is used to address the false negative problem. Experiments on MIMIC-CXR and IU-Xray datasets demonstrate that our framework outperforms state-of-the-art methods in both natural language generation and clinical efficacy metrics.