HEAD-QA: A Healthcare Dataset for Complex Reasoning

David Vilares, Carlos Gómez-Rodríguez


Abstract
We present HEAD-QA, a multi-choice question answering testbed to encourage research on complex reasoning. The questions come from exams to access a specialized position in the Spanish healthcare system, and are challenging even for highly specialized humans. We then consider monolingual (Spanish) and cross-lingual (to English) experiments with information retrieval and neural techniques. We show that: (i) HEAD-QA challenges current methods, and (ii) the results lag well behind human performance, demonstrating its usefulness as a benchmark for future work.
Anthology ID:
P19-1092
Volume:
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2019
Address:
Florence, Italy
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
960–966
Language:
URL:
https://aclanthology.org/P19-1092
DOI:
10.18653/v1/P19-1092
Bibkey:
Cite (ACL):
David Vilares and Carlos Gómez-Rodríguez. 2019. HEAD-QA: A Healthcare Dataset for Complex Reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 960–966, Florence, Italy. Association for Computational Linguistics.
Cite (Informal):
HEAD-QA: A Healthcare Dataset for Complex Reasoning (Vilares & Gómez-Rodríguez, ACL 2019)
Copy Citation:
PDF:
https://aclanthology.org/P19-1092.pdf
Video:
 https://vimeo.com/384467509
Data
HeadQASQuAD