Analysis and Evaluation of Language Models for Word Sense Disambiguation

Daniel Loureiro, Kiamehr Rezaee, Mohammad Taher Pilehvar, Jose Camacho-Collados


Abstract
Transformer-based language models have taken many fields in NLP by storm. BERT and its derivatives dominate most of the existing evaluation benchmarks, including those for Word Sense Disambiguation (WSD), thanks to their ability in capturing context-sensitive semantic nuances. However, there is still little knowledge about their capabilities and potential limitations in encoding and recovering word senses. In this article, we provide an in-depth quantitative and qualitative analysis of the celebrated BERT model with respect to lexical ambiguity. One of the main conclusions of our analysis is that BERT can accurately capture high-level sense distinctions, even when a limited number of examples is available for each word sense. Our analysis also reveals that in some cases language models come close to solving coarse-grained noun disambiguation under ideal conditions in terms of availability of training data and computing resources. However, this scenario rarely occurs in real-world settings and, hence, many practical challenges remain even in the coarse-grained setting. We also perform an in-depth comparison of the two main language model-based WSD strategies, namely, fine-tuning and feature extraction, finding that the latter approach is more robust with respect to sense bias and it can better exploit limited available training data. In fact, the simple feature extraction strategy of averaging contextualized embeddings proves robust even using only three training sentences per word sense, with minimal improvements obtained by increasing the size of this training data.
Anthology ID:
2021.cl-2.14
Volume:
Computational Linguistics, Volume 47, Issue 2 - June 2021
Month:
June
Year:
2021
Address:
Cambridge, MA
Venue:
CL
SIG:
Publisher:
MIT Press
Note:
Pages:
387–443
Language:
URL:
https://aclanthology.org/2021.cl-2.14
DOI:
10.1162/coli_a_00405
Bibkey:
Cite (ACL):
Daniel Loureiro, Kiamehr Rezaee, Mohammad Taher Pilehvar, and Jose Camacho-Collados. 2021. Analysis and Evaluation of Language Models for Word Sense Disambiguation. Computational Linguistics, 47(2):387–443.
Cite (Informal):
Analysis and Evaluation of Language Models for Word Sense Disambiguation (Loureiro et al., CL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.cl-2.14.pdf
Video:
 https://aclanthology.org/2021.cl-2.14.mp4
Code
 danlou/bert-disambiguation
Data
CoarseWSD-20GLUESuperGLUEWord Sense Disambiguation: a Unified Evaluation Framework and Empirical Comparison