Do We Know What We Don’t Know? Studying Unanswerable Questions beyond SQuAD 2.0

Elior Sulem, Jamaal Hay, Dan Roth


Abstract
Understanding when a text snippet does not provide a sought after information is an essential part of natural language utnderstanding. Recent work (SQuAD 2.0; Rajpurkar et al., 2018) has attempted to make some progress in this direction by enriching the SQuAD dataset for the Extractive QA task with unanswerable questions. However, as we show, the performance of a top system trained on SQuAD 2.0 drops considerably in out-of-domain scenarios, limiting its use in practical situations. In order to study this we build an out-of-domain corpus, focusing on simple event-based questions and distinguish between two types of IDK questions: competitive questions, where the context includes an entity of the same type as the expected answer, and simpler, non-competitive questions where there is no entity of the same type in the context. We find that SQuAD 2.0-based models fail even in the case of the simpler questions. We then analyze the similarities and differences between the IDK phenomenon in Extractive QA and the Recognizing Textual Entailments task (RTE; Dagan et al., 2013) and investigate the extent to which the latter can be used to improve the performance.
Anthology ID:
2021.findings-emnlp.385
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2021
Month:
November
Year:
2021
Address:
Punta Cana, Dominican Republic
Editors:
Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
Venue:
Findings
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
4543–4548
Language:
URL:
https://aclanthology.org/2021.findings-emnlp.385
DOI:
10.18653/v1/2021.findings-emnlp.385
Bibkey:
Cite (ACL):
Elior Sulem, Jamaal Hay, and Dan Roth. 2021. Do We Know What We Don’t Know? Studying Unanswerable Questions beyond SQuAD 2.0. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4543–4548, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Do We Know What We Don’t Know? Studying Unanswerable Questions beyond SQuAD 2.0 (Sulem et al., Findings 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.findings-emnlp.385.pdf
Video:
 https://aclanthology.org/2021.findings-emnlp.385.mp4
Data
GLUESQuAD