MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension

Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, Danqi Chen


Abstract
We present the results of the Machine Reading for Question Answering (MRQA) 2019 shared task on evaluating the generalization capabilities of reading comprehension systems. In this task, we adapted and unified 18 distinct question answering datasets into the same format. Among them, six datasets were made available for training, six datasets were made available for development, and the rest were hidden for final evaluation. Ten teams submitted systems, which explored various ideas including data sampling, multi-task learning, adversarial training and ensembling. The best system achieved an average F1 score of 72.5 on the 12 held-out datasets, 10.7 absolute points higher than our initial baseline based on BERT.
Anthology ID:
D19-5801
Volume:
Proceedings of the 2nd Workshop on Machine Reading for Question Answering
Month:
November
Year:
2019
Address:
Hong Kong, China
Venues:
EMNLP | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1–13
Language:
URL:
https://aclanthology.org/D19-5801
DOI:
10.18653/v1/D19-5801
Bibkey:
Copy Citation:
PDF:
https://aclanthology.org/D19-5801.pdf
Code
 mrqa/MRQA-Shared-Task-2019
Data
MRQA 2019DROPDuoRCHotpotQAMCTestNatural QuestionsNewsQAQAMRRACESQuADSearchQATriviaQA