Injecting Wiktionary to improve token-level contextual representations using contrastive learning

Anna Mosolova, Marie Candito, Carlos Ramisch


Abstract
While static word embeddings are blind to context, for lexical semantics tasks context is rather too present in contextual word embeddings, vectors of same-meaning occurrences being too different (Ethayarajh, 2019). Fine-tuning pre-trained language models (PLMs) using contrastive learning was proposed, leveraging automatically self-augmented examples (Liu et al., 2021b). In this paper, we investigate how to inject a lexicon as an alternative source of supervision, using the English Wiktionary. We also test how dimensionality reduction impacts the resulting contextual word embeddings. We evaluate our approach on the Word-In-Context (WiC) task, in the unsupervised setting (not using the training set). We achieve new SoTA result on the original WiC test set. We also propose two new WiC test sets for which we show that our fine-tuning method achieves substantial improvements. We also observe improvements, although modest, for the semantic frame induction task. Although we experimented on English to allow comparison with related work, our method is adaptable to the many languages for which large Wiktionaries exist.
Anthology ID:
2024.eacl-short.5
Volume:
Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)
Month:
March
Year:
2024
Address:
St. Julian’s, Malta
Editors:
Yvette Graham, Matthew Purver
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
34–41
Language:
URL:
https://aclanthology.org/2024.eacl-short.5
DOI:
Bibkey:
Cite (ACL):
Anna Mosolova, Marie Candito, and Carlos Ramisch. 2024. Injecting Wiktionary to improve token-level contextual representations using contrastive learning. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pages 34–41, St. Julian’s, Malta. Association for Computational Linguistics.
Cite (Informal):
Injecting Wiktionary to improve token-level contextual representations using contrastive learning (Mosolova et al., EACL 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.eacl-short.5.pdf
Video:
 https://aclanthology.org/2024.eacl-short.5.mp4