SituatedQA: Incorporating Extra-Linguistic Contexts into QA

Michael Zhang, Eunsol Choi


Abstract
Answers to the same question may change depending on the extra-linguistic contexts (when and where the question was asked). To study this challenge, we introduce SituatedQA, an open-retrieval QA dataset where systems must produce the correct answer to a question given the temporal or geographical context. To construct SituatedQA, we first identify such questions in existing QA datasets. We find that a significant proportion of information seeking questions have context-dependent answers (e.g. roughly 16.5% of NQ-Open). For such context-dependent questions, we then crowdsource alternative contexts and their corresponding answers. Our study shows that existing models struggle with producing answers that are frequently updated or from uncommon locations. We further quantify how existing models, which are trained on data collected in the past, fail to generalize to answering questions asked in the present, even when provided with an updated evidence corpus (a roughly 15 point drop in accuracy). Our analysis suggests that open-retrieval QA benchmarks should incorporate extra-linguistic context to stay relevant globally and in the future. Our data, code, and datasheet are available at https://situatedqa.github.io/.
Anthology ID:
2021.emnlp-main.586
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Editors:
Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
7371–7387
Language:
URL:
https://aclanthology.org/2021.emnlp-main.586
DOI:
10.18653/v1/2021.emnlp-main.586
Bibkey:
Cite (ACL):
Michael Zhang and Eunsol Choi. 2021. SituatedQA: Incorporating Extra-Linguistic Contexts into QA. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7371–7387, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
SituatedQA: Incorporating Extra-Linguistic Contexts into QA (Zhang & Choi, EMNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.emnlp-main.586.pdf
Video:
 https://aclanthology.org/2021.emnlp-main.586.mp4
Data
SituatedQADROPNatural Questions