Robust Question Answering Through Sub-part Alignment

Jifan Chen, Greg Durrett


Abstract
Current textual question answering (QA) models achieve strong performance on in-domain test sets, but often do so by fitting surface-level patterns, so they fail to generalize to out-of-distribution settings. To make a more robust and understandable QA system, we model question answering as an alignment problem. We decompose both the question and context into smaller units based on off-the-shelf semantic representations (here, semantic roles), and align the question to a subgraph of the context in order to find the answer. We formulate our model as a structured SVM, with alignment scores computed via BERT, and we can train end-to-end despite using beam search for approximate inference. Our use of explicit alignments allows us to explore a set of constraints with which we can prohibit certain types of bad model behavior arising in cross-domain settings. Furthermore, by investigating differences in scores across different potential answers, we can seek to understand what particular aspects of the input lead the model to choose the answer without relying on post-hoc explanation techniques. We train our model on SQuAD v1.1 and test it on several adversarial and out-of-domain datasets. The results show that our model is more robust than the standard BERT QA model, and constraints derived from alignment scores allow us to effectively trade off coverage and accuracy.
Anthology ID:
2021.naacl-main.98
Volume:
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
June
Year:
2021
Address:
Online
Editors:
Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, Yichao Zhou
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1251–1263
Language:
URL:
https://aclanthology.org/2021.naacl-main.98
DOI:
10.18653/v1/2021.naacl-main.98
Bibkey:
Cite (ACL):
Jifan Chen and Greg Durrett. 2021. Robust Question Answering Through Sub-part Alignment. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1251–1263, Online. Association for Computational Linguistics.
Cite (Informal):
Robust Question Answering Through Sub-part Alignment (Chen & Durrett, NAACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.naacl-main.98.pdf
Video:
 https://aclanthology.org/2021.naacl-main.98.mp4
Data
BioMRQANewsQASQuAD