An Analysis of Negation in Natural Language Understanding Corpora

Md Mosharaf Hossain, Dhivya Chinnappa, Eduardo Blanco


Abstract
This paper analyzes negation in eight popular corpora spanning six natural language understanding tasks. We show that these corpora have few negations compared to general-purpose English, and that the few negations in them are often unimportant. Indeed, one can often ignore negations and still make the right predictions. Additionally, experimental results show that state-of-the-art transformers trained with these corpora obtain substantially worse results with instances that contain negation, especially if the negations are important. We conclude that new corpora accounting for negation are needed to solve natural language understanding tasks when negation is present.
Anthology ID:
2022.acl-short.81
Volume:
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
Month:
May
Year:
2022
Address:
Dublin, Ireland
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
716–723
Language:
URL:
https://aclanthology.org/2022.acl-short.81
DOI:
10.18653/v1/2022.acl-short.81
Bibkey:
Cite (ACL):
Md Mosharaf Hossain, Dhivya Chinnappa, and Eduardo Blanco. 2022. An Analysis of Negation in Natural Language Understanding Corpora. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 716–723, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
An Analysis of Negation in Natural Language Understanding Corpora (Hossain et al., ACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.acl-short.81.pdf
Software:
 2022.acl-short.81.software.zip
Code
 mosharafhossain/negation-and-nlu
Data
COPACommonsenseQAGLUEMultiNLIQNLISNLISSTSuperGLUEWSCWiC