Can NLI Models Verify QA Systems’ Predictions?

Jifan Chen, Eunsol Choi, Greg Durrett


Abstract
To build robust question answering systems, we need the ability to verify whether answers to questions are truly correct, not just “good enough” in the context of imperfect QA datasets. We explore the use of natural language inference (NLI) as a way to achieve this goal, as NLI inherently requires the premise (document context) to contain all necessary information to support the hypothesis (proposed answer to the question). We leverage large pre-trained models and recent prior datasets to construct powerful question conversion and decontextualization modules, which can reformulate QA instances as premise-hypothesis pairs with very high reliability. Then, by combining standard NLI datasets with NLI examples automatically derived from QA training data, we can train NLI models to evaluate QA models’ proposed answers. We show that our approach improves the confidence estimation of a QA model across different domains, evaluated in a selective QA setting. Careful manual analysis over the predictions of our NLI model shows that it can further identify cases where the QA model produces the right answer for the wrong reason, i.e., when the answer sentence cannot address all aspects of the question.
Anthology ID:
2021.findings-emnlp.324
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2021
Month:
November
Year:
2021
Address:
Punta Cana, Dominican Republic
Editors:
Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
Venue:
Findings
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
3841–3854
Language:
URL:
https://aclanthology.org/2021.findings-emnlp.324
DOI:
10.18653/v1/2021.findings-emnlp.324
Bibkey:
Cite (ACL):
Jifan Chen, Eunsol Choi, and Greg Durrett. 2021. Can NLI Models Verify QA Systems’ Predictions?. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3841–3854, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Can NLI Models Verify QA Systems’ Predictions? (Chen et al., Findings 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.findings-emnlp.324.pdf
Video:
 https://aclanthology.org/2021.findings-emnlp.324.mp4
Code
 jifan-chen/qa-verification-via-nli
Data
BioASQFEVERMRQAMultiNLINatural QuestionsSQuADTriviaQA