More Bang for Your Buck: Natural Perturbation for Robust Question Answering

Daniel Khashabi, Tushar Khot, Ashish Sabharwal


Abstract
Deep learning models for linguistic tasks require large training datasets, which are expensive to create. As an alternative to the traditional approach of creating new instances by repeating the process of creating one instance, we propose doing so by first collecting a set of seed examples and then applying human-driven natural perturbations (as opposed to rule-based machine perturbations), which often change the gold label as well. Such perturbations have the advantage of being relatively easier (and hence cheaper) to create than writing out completely new examples. Further, they help address the issue that even models achieving human-level scores on NLP datasets are known to be considerably sensitive to small changes in input. To evaluate the idea, we consider a recent question-answering dataset (BOOLQ) and study our approach as a function of the perturbation cost ratio, the relative cost of perturbing an existing question vs. creating a new one from scratch. We find that when natural perturbations are moderately cheaper to create (cost ratio under 60%), it is more effective to use them for training BOOLQ models: such models exhibit 9% higher robustness and 4.5% stronger generalization, while retaining performance on the original BOOLQ dataset.
Anthology ID:
2020.emnlp-main.12
Volume:
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Month:
November
Year:
2020
Address:
Online
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
163–170
Language:
URL:
https://aclanthology.org/2020.emnlp-main.12
DOI:
10.18653/v1/2020.emnlp-main.12
Bibkey:
Cite (ACL):
Daniel Khashabi, Tushar Khot, and Ashish Sabharwal. 2020. More Bang for Your Buck: Natural Perturbation for Robust Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 163–170, Online. Association for Computational Linguistics.
Cite (Informal):
More Bang for Your Buck: Natural Perturbation for Robust Question Answering (Khashabi et al., EMNLP 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.emnlp-main.12.pdf
Video:
 https://slideslive.com/38939179