InferES : A Natural Language Inference Corpus for Spanish Featuring Negation-Based Contrastive and Adversarial Examples

Venelin Kovatchev, Mariona Taulé


Abstract
In this paper we present InferES - an original corpus for Natural Language Inference (NLI) in European Spanish. We propose, implement, and analyze a variety of corpus-creating strategies utilizing expert linguists and crowd workers. The objectives behind InferES are to provide high-quality data, and at the same time to facilitate the systematic evaluation of automated systems. Specifically, we focus on measuring and improving the performance of machine learning systems on negation-based adversarial examples and their ability to generalize across out-of-distribution topics. We train two transformer models on InferES (8,055 gold examples) in a variety of scenarios. Our best model obtains 72.8% accuracy, leaving a lot of room for improvement. The “hypothesis-only” baseline performs only 2%-5% higher than majority, indicating much fewer annotation artifacts than prior work. We show that models trained on InferES generalize very well across topics (both in- and out-of-distribution) and perform moderately well on negation-based adversarial examples.
Anthology ID:
2022.coling-1.340
Volume:
Proceedings of the 29th International Conference on Computational Linguistics
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
3873–3884
Language:
URL:
https://aclanthology.org/2022.coling-1.340
DOI:
Bibkey:
Cite (ACL):
Venelin Kovatchev and Mariona Taulé. 2022. InferES : A Natural Language Inference Corpus for Spanish Featuring Negation-Based Contrastive and Adversarial Examples. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3873–3884, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Cite (Informal):
InferES : A Natural Language Inference Corpus for Spanish Featuring Negation-Based Contrastive and Adversarial Examples (Kovatchev & Taulé, COLING 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.coling-1.340.pdf
Code
 venelink/inferes
Data
GLUEMultiNLISNLISuperGLUEXNLI