Code-switched Language Models Using Dual RNNs and Same-Source Pretraining

Saurabh Garg, Tanmay Parekh, Preethi Jyothi


Abstract
This work focuses on building language models (LMs) for code-switched text. We propose two techniques that significantly improve these LMs: 1) A novel recurrent neural network unit with dual components that focus on each language in the code-switched text separately 2) Pretraining the LM using synthetic text from a generative model estimated using the training data. We demonstrate the effectiveness of our proposed techniques by reporting perplexities on a Mandarin-English task and derive significant reductions in perplexity.
Anthology ID:
D18-1346
Volume:
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
Month:
October-November
Year:
2018
Address:
Brussels, Belgium
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
3078–3083
Language:
URL:
https://aclanthology.org/D18-1346
DOI:
10.18653/v1/D18-1346
Bibkey:
Cite (ACL):
Saurabh Garg, Tanmay Parekh, and Preethi Jyothi. 2018. Code-switched Language Models Using Dual RNNs and Same-Source Pretraining. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3078–3083, Brussels, Belgium. Association for Computational Linguistics.
Cite (Informal):
Code-switched Language Models Using Dual RNNs and Same-Source Pretraining (Garg et al., EMNLP 2018)
Copy Citation:
PDF:
https://aclanthology.org/D18-1346.pdf