Shortcutted Commonsense: Data Spuriousness in Deep Learning of Commonsense Reasoning

Ruben Branco, António Branco, João António Rodrigues, João Ricardo Silva


Abstract
Commonsense is a quintessential human capacity that has been a core challenge to Artificial Intelligence since its inception. Impressive results in Natural Language Processing tasks, including in commonsense reasoning, have consistently been achieved with Transformer neural language models, even matching or surpassing human performance in some benchmarks. Recently, some of these advances have been called into question: so called data artifacts in the training data have been made evident as spurious correlations and shallow shortcuts that in some cases are leveraging these outstanding results. In this paper we seek to further pursue this analysis into the realm of commonsense related language processing tasks. We undertake a study on different prominent benchmarks that involve commonsense reasoning, along a number of key stress experiments, thus seeking to gain insight on whether the models are learning transferable generalizations intrinsic to the problem at stake or just taking advantage of incidental shortcuts in the data items. The results obtained indicate that most datasets experimented with are problematic, with models resorting to non-robust features and appearing not to be learning and generalizing towards the overall tasks intended to be conveyed or exemplified by the datasets.
Anthology ID:
2021.emnlp-main.113
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1504–1521
Language:
URL:
https://aclanthology.org/2021.emnlp-main.113
DOI:
10.18653/v1/2021.emnlp-main.113
Bibkey:
Copy Citation:
PDF:
https://aclanthology.org/2021.emnlp-main.113.pdf
Software:
 2021.emnlp-main.113.Software.zip
Code
 nlx-group/shortcutted-commonsense-reasoning
Data
ARCCommonsenseQADROPPIQASuperGLUEWebText