Dynamic Sentence Sampling for Efficient Training of Neural Machine Translation

Rui Wang, Masao Utiyama, Eiichiro Sumita


Abstract
Traditional Neural machine translation (NMT) involves a fixed training procedure where each sentence is sampled once during each epoch. In reality, some sentences are well-learned during the initial few epochs; however, using this approach, the well-learned sentences would continue to be trained along with those sentences that were not well learned for 10-30 epochs, which results in a wastage of time. Here, we propose an efficient method to dynamically sample the sentences in order to accelerate the NMT training. In this approach, a weight is assigned to each sentence based on the measured difference between the training costs of two iterations. Further, in each epoch, a certain percentage of sentences are dynamically sampled according to their weights. Empirical results based on the NIST Chinese-to-English and the WMT English-to-German tasks show that the proposed method can significantly accelerate the NMT training and improve the NMT performance.
Anthology ID:
P18-2048
Volume:
Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
Month:
July
Year:
2018
Address:
Melbourne, Australia
Editors:
Iryna Gurevych, Yusuke Miyao
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
298–304
Language:
URL:
https://aclanthology.org/P18-2048
DOI:
10.18653/v1/P18-2048
Bibkey:
Cite (ACL):
Rui Wang, Masao Utiyama, and Eiichiro Sumita. 2018. Dynamic Sentence Sampling for Efficient Training of Neural Machine Translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 298–304, Melbourne, Australia. Association for Computational Linguistics.
Cite (Informal):
Dynamic Sentence Sampling for Efficient Training of Neural Machine Translation (Wang et al., ACL 2018)
Copy Citation:
PDF:
https://aclanthology.org/P18-2048.pdf
Poster:
 P18-2048.Poster.pdf