Effective Crowd-Annotation of Participants, Interventions, and Outcomes in the Text of Clinical Trial Reports

Markus Zlabinger, Marta Sabou, Sebastian Hofstätter, Allan Hanbury


Abstract
The search for Participants, Interventions, and Outcomes (PIO) in clinical trial reports is a critical task in Evidence Based Medicine. For an automatic PIO extraction, high-quality corpora are needed. Obtaining such a corpus from crowdworkers, however, has been shown to be ineffective since (i) workers usually lack domain-specific expertise to conduct the task with sufficient quality, and (ii) the standard approach of annotating entire abstracts of trial reports as one task-instance (i.e. HIT) leads to an uneven distribution in task effort. In this paper, we switch from entire abstract to sentence annotation, referred to as the SenBase approach. We build upon SenBase in SenSupport, where we compensate the lack of domain-specific expertise of crowdworkers by showing for each task-instance similar sentences that are already annotated by experts. Such tailored task-instance examples are retrieved via unsupervised semantic short-text similarity (SSTS) method – and we evaluate nine methods to find an effective solution for SenSupport. We compute the Cohen’s Kappa agreement between crowd-annotations and gold standard annotations and show that (i) both sentence-based approaches outperform a Baseline approach where entire abstracts are annotated; (ii) supporting annotators with tailored task-instance examples is the best performing approach with Kappa agreements of 0.78/0.75/0.69 for P, I, and O respectively.
Anthology ID:
2020.findings-emnlp.274
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2020
Month:
November
Year:
2020
Address:
Online
Editors:
Trevor Cohn, Yulan He, Yang Liu
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3064–3074
Language:
URL:
https://aclanthology.org/2020.findings-emnlp.274
DOI:
10.18653/v1/2020.findings-emnlp.274
Bibkey:
Cite (ACL):
Markus Zlabinger, Marta Sabou, Sebastian Hofstätter, and Allan Hanbury. 2020. Effective Crowd-Annotation of Participants, Interventions, and Outcomes in the Text of Clinical Trial Reports. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3064–3074, Online. Association for Computational Linguistics.
Cite (Informal):
Effective Crowd-Annotation of Participants, Interventions, and Outcomes in the Text of Clinical Trial Reports (Zlabinger et al., Findings 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.findings-emnlp.274.pdf
Optional supplementary material:
 2020.findings-emnlp.274.OptionalSupplementaryMaterial.zip
Code
 Markus-Zlabinger/pico-annotation +  additional community code
Data
BIOSSESEBM-NLPSNLI