KGVL-BART: Knowledge Graph Augmented Visual Language BART for Radiology Report Generation

Kaveri Kale, Pushpak Bhattacharyya, Milind Gune, Aditya Shetty, Rustom Lawyer


Abstract
Timely generation of radiology reports and diagnoses is a challenge worldwide due to the enormous number of cases and shortage of radiology specialists. In this paper, we propose a Knowledge Graph Augmented Vision Language BART (KGVL-BART) model that takes as input two chest X-ray images- one frontal and the other lateral- along with tags which are diagnostic keywords, and outputs a report with the patient-specific findings. Our system development effort is divided into 3 stages: i) construction of the Chest X-ray KG (referred to as chestX-KG), ii) image feature extraction, and iii) training a KGVL-BART model using the visual, text, and KG data. The dataset we use is the well-known Indiana University Chest X-ray reports with the train, validation, and test split of 3025 instances, 300 instances, and 500 instances respectively. We construct a Chest X-Ray knowledge graph from these reports by extracting entity1-relation-entity2 triples; the triples get extracted by a rule-based tool of our own. Constructed KG is verified by two experienced radiologists (with experience of 30 years and 8 years, respectively). We demonstrate that our model- KGVL-BART- outperforms State-of-the-Art transformer-based models on standard NLG scoring metrics. We also include a qualitative evaluation of our system by experienced radiologist (with experience of 30 years) on the test data, which showed that 73% of the reports generated were fully correct, only 5.5% are completely wrong and 21.5% have important missing details though overall correct. To the best of our knowledge, ours is the first system to make use of multi-modality and domain knowledge to generate X-ray reports automatically.
Anthology ID:
2023.eacl-main.246
Volume:
Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics
Month:
May
Year:
2023
Address:
Dubrovnik, Croatia
Editors:
Andreas Vlachos, Isabelle Augenstein
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3401–3411
Language:
URL:
https://aclanthology.org/2023.eacl-main.246
DOI:
10.18653/v1/2023.eacl-main.246
Bibkey:
Cite (ACL):
Kaveri Kale, Pushpak Bhattacharyya, Milind Gune, Aditya Shetty, and Rustom Lawyer. 2023. KGVL-BART: Knowledge Graph Augmented Visual Language BART for Radiology Report Generation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3401–3411, Dubrovnik, Croatia. Association for Computational Linguistics.
Cite (Informal):
KGVL-BART: Knowledge Graph Augmented Visual Language BART for Radiology Report Generation (Kale et al., EACL 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.eacl-main.246.pdf
Video:
 https://aclanthology.org/2023.eacl-main.246.mp4