Daniel C. Castro


2024

pdf bib
MAIRA at RRG24: A specialised large multimodal model for radiology report generation
Shaury Srivastav | Mercy Ranjit | Fernando Pérez-García | Kenza Bouzid | Shruthi Bannur | Daniel C. Castro | Anton Schwaighofer | Harshita Sharma | Maximilian Ilse | Valentina Salvatelli | Sam Bond-Taylor | Fabian Falck | Anja Thieme | Hannah Richardson | Matthew P. Lungren | Stephanie L. Hyland | Javier Alvarez-Valle
Proceedings of the 23rd Workshop on Biomedical Natural Language Processing

This paper discusses the participation of the MSR MAIRA team in the Large-Scale Radiology Report Generation Shared Task Challenge, as part of the BioNLP workshop at ACL 2024. We present a radiology-specific multimodal model designed to generate radiological reports from chest X-Rays (CXRs). Our proposed model combines a CXR-specific image encoder RAD-DINO with a Large Language Model (LLM) based on Vicuna-7B, via a multi-layer perceptron (MLP) adapter. Both the adapter and the LLM have been fine-tuned in a single-stage training setup to generate radiology reports. Experimental results indicate that a joint training setup with findings and impression sections improves findings prediction. Additionally, incorporating lateral images alongside frontal images when available further enhances all metrics. More information and resources about MAIRA can be found on the project website: http://aka.ms/maira.