Improving Character-Aware Neural Language Model by Warming up Character Encoder under Skip-gram Architecture

Yukun Feng, Chenlong Hu, Hidetaka Kamigaito, Hiroya Takamura, Manabu Okumura


Abstract
Character-aware neural language models can capture the relationship between words by exploiting character-level information and are particularly effective for languages with rich morphology. However, these models are usually biased towards information from surface forms. To alleviate this problem, we propose a simple and effective method to improve a character-aware neural language model by forcing a character encoder to produce word-based embeddings under Skip-gram architecture in a warm-up step without extra training data. We empirically show that the resulting character-aware neural language model achieves obvious improvements of perplexity scores on typologically diverse languages, that contain many low-frequency or unseen words.
Anthology ID:
2021.ranlp-1.48
Volume:
Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)
Month:
September
Year:
2021
Address:
Held Online
Editors:
Ruslan Mitkov, Galia Angelova
Venue:
RANLP
SIG:
Publisher:
INCOMA Ltd.
Note:
Pages:
421–427
Language:
URL:
https://aclanthology.org/2021.ranlp-1.48
DOI:
Bibkey:
Cite (ACL):
Yukun Feng, Chenlong Hu, Hidetaka Kamigaito, Hiroya Takamura, and Manabu Okumura. 2021. Improving Character-Aware Neural Language Model by Warming up Character Encoder under Skip-gram Architecture. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 421–427, Held Online. INCOMA Ltd..
Cite (Informal):
Improving Character-Aware Neural Language Model by Warming up Character Encoder under Skip-gram Architecture (Feng et al., RANLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.ranlp-1.48.pdf