Evaluating the Robustness of Adverse Drug Event Classification Models using Templates

Dorothea MacPhail, David Harbecke, Lisa Raithel, Sebastian Möller


Abstract
An adverse drug effect (ADE) is any harmful event resulting from medical drug treatment. Despite their importance, ADEs are often under-reported in official channels. Some research has therefore turned to detecting discussions of ADEs in social media. Impressive results have been achieved in various attempts to detect ADEs. In a high-stakes domain such as medicine, however, an in-depth evaluation of a model’s abilities is crucial. We address the issue of thorough performance evaluation in detecting ADEs with hand-crafted templates for four capabilities, temporal order, negation, sentiment and beneficial effect. We find that models with similar performance on held-out test sets have varying results on these capabilities.
Anthology ID:
2024.bionlp-1.3
Volume:
Proceedings of the 23rd Workshop on Biomedical Natural Language Processing
Month:
August
Year:
2024
Address:
Bangkok, Thailand
Editors:
Dina Demner-Fushman, Sophia Ananiadou, Makoto Miwa, Kirk Roberts, Junichi Tsujii
Venues:
BioNLP | WS
SIG:
SIGBIOMED
Publisher:
Association for Computational Linguistics
Note:
Pages:
25–38
Language:
URL:
https://aclanthology.org/2024.bionlp-1.3
DOI:
10.18653/v1/2024.bionlp-1.3
Bibkey:
Cite (ACL):
Dorothea MacPhail, David Harbecke, Lisa Raithel, and Sebastian Möller. 2024. Evaluating the Robustness of Adverse Drug Event Classification Models using Templates. In Proceedings of the 23rd Workshop on Biomedical Natural Language Processing, pages 25–38, Bangkok, Thailand. Association for Computational Linguistics.
Cite (Informal):
Evaluating the Robustness of Adverse Drug Event Classification Models using Templates (MacPhail et al., BioNLP-WS 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.bionlp-1.3.pdf