Sparsity Makes Sense: Word Sense Disambiguation Using Sparse Contextualized Word Representations

Gábor Berend


Abstract
In this paper, we demonstrate that by utilizing sparse word representations, it becomes possible to surpass the results of more complex task-specific models on the task of fine-grained all-words word sense disambiguation. Our proposed algorithm relies on an overcomplete set of semantic basis vectors that allows us to obtain sparse contextualized word representations. We introduce such an information theory-inspired synset representation based on the co-occurrence of word senses and non-zero coordinates for word forms which allows us to achieve an aggregated F-score of 78.8 over a combination of five standard word sense disambiguating benchmark datasets. We also demonstrate the general applicability of our proposed framework by evaluating it towards part-of-speech tagging on four different treebanks. Our results indicate a significant improvement over the application of the dense word representations.
Anthology ID:
2020.emnlp-main.683
Volume:
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Month:
November
Year:
2020
Address:
Online
Editors:
Bonnie Webber, Trevor Cohn, Yulan He, Yang Liu
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
8498–8508
Language:
URL:
https://aclanthology.org/2020.emnlp-main.683
DOI:
10.18653/v1/2020.emnlp-main.683
Bibkey:
Cite (ACL):
Gábor Berend. 2020. Sparsity Makes Sense: Word Sense Disambiguation Using Sparse Contextualized Word Representations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8498–8508, Online. Association for Computational Linguistics.
Cite (Informal):
Sparsity Makes Sense: Word Sense Disambiguation Using Sparse Contextualized Word Representations (Berend, EMNLP 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.emnlp-main.683.pdf
Video:
 https://slideslive.com/38939289
Data
Word Sense Disambiguation: a Unified Evaluation Framework and Empirical Comparison