Stress Test Evaluation of Biomedical Word Embeddings

Vladimir Araujo, Andrés Carvallo, Carlos Aspillaga, Camilo Thorne, Denis Parra


Abstract
The success of pretrained word embeddings has motivated their use in the biomedical domain, with contextualized embeddings yielding remarkable results in several biomedical NLP tasks. However, there is a lack of research on quantifying their behavior under severe “stress” scenarios. In this work, we systematically evaluate three language models with adversarial examples – automatically constructed tests that allow us to examine how robust the models are. We propose two types of stress scenarios focused on the biomedical named entity recognition (NER) task, one inspired by spelling errors and another based on the use of synonyms for medical terms. Our experiments with three benchmarks show that the performance of the original models decreases considerably, in addition to revealing their weaknesses and strengths. Finally, we show that adversarial training causes the models to improve their robustness and even to exceed the original performance in some cases.
Anthology ID:
2021.bionlp-1.13
Volume:
Proceedings of the 20th Workshop on Biomedical Language Processing
Month:
June
Year:
2021
Address:
Online
Editors:
Dina Demner-Fushman, Kevin Bretonnel Cohen, Sophia Ananiadou, Junichi Tsujii
Venue:
BioNLP
SIG:
SIGBIOMED
Publisher:
Association for Computational Linguistics
Note:
Pages:
119–125
Language:
URL:
https://aclanthology.org/2021.bionlp-1.13
DOI:
10.18653/v1/2021.bionlp-1.13
Bibkey:
Cite (ACL):
Vladimir Araujo, Andrés Carvallo, Carlos Aspillaga, Camilo Thorne, and Denis Parra. 2021. Stress Test Evaluation of Biomedical Word Embeddings. In Proceedings of the 20th Workshop on Biomedical Language Processing, pages 119–125, Online. Association for Computational Linguistics.
Cite (Informal):
Stress Test Evaluation of Biomedical Word Embeddings (Araujo et al., BioNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.bionlp-1.13.pdf
Code
 ialab-puc/BioNLP-StressTest
Data
BC5CDRBLUENCBI Disease