Towards Developing a Multilingual and Code-Mixed Visual Question Answering System by Knowledge Distillation

Humair Raj Khan, Deepak Gupta, Asif Ekbal


Abstract
Pre-trained language-vision models have shown remarkable performance on the visual question answering (VQA) task. However, most pre-trained models are trained by only considering monolingual learning, especially the resource-rich language like English. Training such models for multilingual setups demand high computing resources and multilingual language-vision dataset which hinders their application in practice. To alleviate these challenges, we propose a knowledge distillation approach to extend an English language-vision model (teacher) into an equally effective multilingual and code-mixed model (student). Unlike the existing knowledge distillation methods, which only use the output from the last layer of the teacher network for distillation, our student model learns and imitates the teacher from multiple intermediate layers (language and vision encoders) with appropriately designed distillation objectives for incremental knowledge extraction. We also create the large-scale multilingual and code-mixed VQA dataset in eleven different language setups considering the multiple Indian and European languages. Experimental results and in-depth analysis show the effectiveness of the proposed VQA model over the pre-trained language-vision models on eleven diverse language setups.
Anthology ID:
2021.findings-emnlp.151
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2021
Month:
November
Year:
2021
Address:
Punta Cana, Dominican Republic
Editors:
Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
Venue:
Findings
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
1753–1767
Language:
URL:
https://aclanthology.org/2021.findings-emnlp.151
DOI:
10.18653/v1/2021.findings-emnlp.151
Bibkey:
Cite (ACL):
Humair Raj Khan, Deepak Gupta, and Asif Ekbal. 2021. Towards Developing a Multilingual and Code-Mixed Visual Question Answering System by Knowledge Distillation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1753–1767, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Towards Developing a Multilingual and Code-Mixed Visual Question Answering System by Knowledge Distillation (Raj Khan et al., Findings 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.findings-emnlp.151.pdf
Video:
 https://aclanthology.org/2021.findings-emnlp.151.mp4
Data
MuCo-VQAMCVQAVisual Question Answering