Visual Question Generation from Radiology Images

Mourad Sarrouti, Asma Ben Abacha, Dina Demner-Fushman


Abstract
Visual Question Generation (VQG), the task of generating a question based on image contents, is an increasingly important area that combines natural language processing and computer vision. Although there are some recent works that have attempted to generate questions from images in the open domain, the task of VQG in the medical domain has not been explored so far. In this paper, we introduce an approach to generation of visual questions about radiology images called VQGR, i.e. an algorithm that is able to ask a question when shown an image. VQGR first generates new training data from the existing examples, based on contextual word embeddings and image augmentation techniques. It then uses the variational auto-encoders model to encode images into a latent space and decode natural language questions. Experimental automatic evaluations performed on the VQA-RAD dataset of clinical visual questions show that VQGR achieves good performances compared with the baseline system. The source code is available at https://github.com/sarrouti/vqgr.
Anthology ID:
2020.alvr-1.3
Volume:
Proceedings of the First Workshop on Advances in Language and Vision Research
Month:
July
Year:
2020
Address:
Online
Editors:
Xin Wang, Jesse Thomason, Ronghang Hu, Xinlei Chen, Peter Anderson, Qi Wu, Asli Celikyilmaz, Jason Baldridge, William Yang Wang
Venue:
ALVR
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
12–18
Language:
URL:
https://aclanthology.org/2020.alvr-1.3
DOI:
10.18653/v1/2020.alvr-1.3
Bibkey:
Cite (ACL):
Mourad Sarrouti, Asma Ben Abacha, and Dina Demner-Fushman. 2020. Visual Question Generation from Radiology Images. In Proceedings of the First Workshop on Advances in Language and Vision Research, pages 12–18, Online. Association for Computational Linguistics.
Cite (Informal):
Visual Question Generation from Radiology Images (Sarrouti et al., ALVR 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.alvr-1.3.pdf
Video:
 http://slideslive.com/38929760
Code
 sarrouti/vqgr