RedHOT: A Corpus of Annotated Medical Questions, Experiences, and Claims on Social Media

Somin Wadhwa, Vivek Khetan, Silvio Amir, Byron Wallace


Abstract
We present Reddit Health Online Talk (RedHOT), a corpus of 22,000 richly annotated social media posts from Reddit spanning 24 health conditions. Annotations include demarcations of spans corresponding to medical claims, personal experiences, and questions. We collect additional granular annotations on identified claims. Specifically, we mark snippets that describe patient Populations, Interventions, and Outcomes (PIO elements) within these. Using this corpus, we introduce the task of retrieving trustworthy evidence relevant to a given claim made on social media. We propose a new method to automatically derive (noisy) supervision for this task which we use to train a dense retrieval model; this outperforms baseline models. Manual evaluation of retrieval results performed by medical doctors indicate that while our system performance is promising, there is considerable room for improvement. We release all annotations collected (and scripts to assemble the dataset), and all code necessary to reproduce the results in this paper at: https://sominw.com/redhot.
Anthology ID:
2023.findings-eacl.61
Volume:
Findings of the Association for Computational Linguistics: EACL 2023
Month:
May
Year:
2023
Address:
Dubrovnik, Croatia
Editors:
Andreas Vlachos, Isabelle Augenstein
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
809–827
Language:
URL:
https://aclanthology.org/2023.findings-eacl.61
DOI:
10.18653/v1/2023.findings-eacl.61
Bibkey:
Cite (ACL):
Somin Wadhwa, Vivek Khetan, Silvio Amir, and Byron Wallace. 2023. RedHOT: A Corpus of Annotated Medical Questions, Experiences, and Claims on Social Media. In Findings of the Association for Computational Linguistics: EACL 2023, pages 809–827, Dubrovnik, Croatia. Association for Computational Linguistics.
Cite (Informal):
RedHOT: A Corpus of Annotated Medical Questions, Experiences, and Claims on Social Media (Wadhwa et al., Findings 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.findings-eacl.61.pdf
Video:
 https://aclanthology.org/2023.findings-eacl.61.mp4