SciNLI: A Corpus for Natural Language Inference on Scientific Text

Mobashir Sadat, Cornelia Caragea


Abstract
Existing Natural Language Inference (NLI) datasets, while being instrumental in the advancement of Natural Language Understanding (NLU) research, are not related to scientific text. In this paper, we introduce SciNLI, a large dataset for NLI that captures the formality in scientific text and contains 107,412 sentence pairs extracted from scholarly papers on NLP and computational linguistics. Given that the text used in scientific literature differs vastly from the text used in everyday language both in terms of vocabulary and sentence structure, our dataset is well suited to serve as a benchmark for the evaluation of scientific NLU models. Our experiments show that SciNLI is harder to classify than the existing NLI datasets. Our best performing model with XLNet achieves a Macro F1 score of only 78.18% and an accuracy of 78.23% showing that there is substantial room for improvement.
Anthology ID:
2022.acl-long.511
Volume:
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Smaranda Muresan, Preslav Nakov, Aline Villavicencio
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
7399–7409
Language:
URL:
https://aclanthology.org/2022.acl-long.511
DOI:
10.18653/v1/2022.acl-long.511
Bibkey:
Cite (ACL):
Mobashir Sadat and Cornelia Caragea. 2022. SciNLI: A Corpus for Natural Language Inference on Scientific Text. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7399–7409, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
SciNLI: A Corpus for Natural Language Inference on Scientific Text (Sadat & Caragea, ACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.acl-long.511.pdf
Code
 msadat3/scinli
Data
ANLIGLUEMultiNLISICKSNLISWAG