Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikes

Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikes Kosuke Nishida author Kyosuke Nishida author Kuniko Saito author 2024-11 text Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing Yaser Al-Onaizan editor Mohit Bansal editor Yun-Nung Chen editor Association for Computational Linguistics Miami, Florida, USA conference publication nishida-etal-2024-initialization 10.18653/v1/2024.emnlp-main.1264 https://aclanthology.org/2024.emnlp-main.1264/ 2024-11 22699 22714