Investigating Negation in Pre-trained Vision-and-language Models

Radina Dobreva, Frank Keller


Abstract
Pre-trained vision-and-language models have achieved impressive results on a variety of tasks, including ones that require complex reasoning beyond object recognition. However, little is known about how they achieve these results or what their limitations are. In this paper, we focus on a particular linguistic capability, namely the understanding of negation. We borrow techniques from the analysis of language models to investigate the ability of pre-trained vision-and-language models to handle negation. We find that these models severely underperform in the presence of negation.
Anthology ID:
2021.blackboxnlp-1.27
Volume:
Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP
Month:
November
Year:
2021
Address:
Punta Cana, Dominican Republic
Editors:
Jasmijn Bastings, Yonatan Belinkov, Emmanuel Dupoux, Mario Giulianelli, Dieuwke Hupkes, Yuval Pinter, Hassan Sajjad
Venue:
BlackboxNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
350–362
Language:
URL:
https://aclanthology.org/2021.blackboxnlp-1.27
DOI:
10.18653/v1/2021.blackboxnlp-1.27
Bibkey:
Cite (ACL):
Radina Dobreva and Frank Keller. 2021. Investigating Negation in Pre-trained Vision-and-language Models. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 350–362, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Investigating Negation in Pre-trained Vision-and-language Models (Dobreva & Keller, BlackboxNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.blackboxnlp-1.27.pdf
Code
 radidd/vision-and-language-negation