Have You Seen That Number? Investigating Extrapolation in Question Answering Models

Jeonghwan Kim, Giwon Hong, Kyung-min Kim, Junmo Kang, Sung-Hyon Myaeng


Abstract
Numerical reasoning in machine reading comprehension (MRC) has shown drastic improvements over the past few years. While the previous models for numerical MRC are able to interpolate the learned numerical reasoning capabilities, it is not clear whether they can perform just as well on numbers unseen in the training dataset. Our work rigorously tests state-of-the-art models on DROP, a numerical MRC dataset, to see if they can handle passages that contain out-of-range numbers. One of the key findings is that the models fail to extrapolate to unseen numbers. Presenting numbers as digit-by-digit input to the model, we also propose the E-digit number form that alleviates the lack of extrapolation in models and reveals the need to treat numbers differently from regular words in the text. Our work provides a valuable insight into the numerical MRC models and the way to represent number forms in MRC.
Anthology ID:
2021.emnlp-main.563
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
7031–7037
Language:
URL:
https://aclanthology.org/2021.emnlp-main.563
DOI:
10.18653/v1/2021.emnlp-main.563
Bibkey:
Cite (ACL):
Jeonghwan Kim, Giwon Hong, Kyung-min Kim, Junmo Kang, and Sung-Hyon Myaeng. 2021. Have You Seen That Number? Investigating Extrapolation in Question Answering Models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7031–7037, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Have You Seen That Number? Investigating Extrapolation in Question Answering Models (Kim et al., EMNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.emnlp-main.563.pdf
Video:
 https://aclanthology.org/2021.emnlp-main.563.mp4
Data
DROP