Clinical Reading Comprehension: A Thorough Analysis of the emrQA Dataset

Xiang Yue, Bernal Jimenez Gutierrez, Huan Sun


Abstract
Machine reading comprehension has made great progress in recent years owing to large-scale annotated datasets. In the clinical domain, however, creating such datasets is quite difficult due to the domain expertise required for annotation. Recently, Pampari et al. (EMNLP’18) tackled this issue by using expert-annotated question templates and existing i2b2 annotations to create emrQA, the first large-scale dataset for question answering (QA) based on clinical notes. In this paper, we provide an in-depth analysis of this dataset and the clinical reading comprehension (CliniRC) task. From our qualitative analysis, we find that (i) emrQA answers are often incomplete, and (ii) emrQA questions are often answerable without using domain knowledge. From our quantitative experiments, surprising results include that (iii) using a small sampled subset (5%-20%), we can obtain roughly equal performance compared to the model trained on the entire dataset, (iv) this performance is close to human expert’s performance, and (v) BERT models do not beat the best performing base model. Following our analysis of the emrQA, we further explore two desired aspects of CliniRC systems: the ability to utilize clinical domain knowledge and to generalize to unseen questions and contexts. We argue that both should be considered when creating future datasets.
Anthology ID:
2020.acl-main.410
Volume:
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2020
Address:
Online
Editors:
Dan Jurafsky, Joyce Chai, Natalie Schluter, Joel Tetreault
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4474–4486
Language:
URL:
https://aclanthology.org/2020.acl-main.410
DOI:
10.18653/v1/2020.acl-main.410
Bibkey:
Cite (ACL):
Xiang Yue, Bernal Jimenez Gutierrez, and Huan Sun. 2020. Clinical Reading Comprehension: A Thorough Analysis of the emrQA Dataset. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4474–4486, Online. Association for Computational Linguistics.
Cite (Informal):
Clinical Reading Comprehension: A Thorough Analysis of the emrQA Dataset (Yue et al., ACL 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.acl-main.410.pdf
Video:
 http://slideslive.com/38929136
Code
 xiangyue9607/CliniRC
Data
SQuADemrQA