On the Interaction of Regularization Factors in Low-resource Neural Machine Translation

Àlex R. Atrio, Andrei Popescu-Belis


Abstract
We explore the roles and interactions of the hyper-parameters governing regularization, and propose a range of values applicable to low-resource neural machine translation. We demonstrate that default or recommended values for high-resource settings are not optimal for low-resource ones, and that more aggressive regularization is needed when resources are scarce, in proportion to their scarcity. We explain our observations by the generalization abilities of sharp vs. flat basins in the loss landscape of a neural network. Results for four regularization factors corroborate our claim: batch size, learning rate, dropout rate, and gradient clipping. Moreover, we show that optimal results are obtained when using several of these factors, and that our findings generalize across datasets of different sizes and languages.
Anthology ID:
2022.eamt-1.14
Volume:
Proceedings of the 23rd Annual Conference of the European Association for Machine Translation
Month:
June
Year:
2022
Address:
Ghent, Belgium
Editors:
Helena Moniz, Lieve Macken, Andrew Rufener, Loïc Barrault, Marta R. Costa-jussà, Christophe Declercq, Maarit Koponen, Ellie Kemp, Spyridon Pilos, Mikel L. Forcada, Carolina Scarton, Joachim Van den Bogaert, Joke Daems, Arda Tezcan, Bram Vanroy, Margot Fonteyne
Venue:
EAMT
SIG:
Publisher:
European Association for Machine Translation
Note:
Pages:
111–120
Language:
URL:
https://aclanthology.org/2022.eamt-1.14
DOI:
Bibkey:
Cite (ACL):
Àlex R. Atrio and Andrei Popescu-Belis. 2022. On the Interaction of Regularization Factors in Low-resource Neural Machine Translation. In Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, pages 111–120, Ghent, Belgium. European Association for Machine Translation.
Cite (Informal):
On the Interaction of Regularization Factors in Low-resource Neural Machine Translation (Atrio & Popescu-Belis, EAMT 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.eamt-1.14.pdf
Code
 alexratrio/reg-factors