Leitner-Guided Memory Replay for Cross-lingual Continual Learning

Meryem M’hamdi, Jonathan May


Abstract
Cross-lingual continual learning aims to continuously fine-tune a downstream model on emerging data from new languages. One major challenge in cross-lingual continual learning is catastrophic forgetting: a stability-plasticity dilemma, where performance on previously seen languages decreases as the model learns to transfer to new languages. Experience replay, which revisits data from a fixed-size memory of old languages while training on new ones, is among the most successful approaches for solving this dilemma. Faced with the challenge of dynamically storing the memory with high-quality examples while complying with its fixed size limitations, we consider Leitner queuing, a human-inspired spaced-repetition technique, to determine what should be replayed at each phase of learning. Via a controlled set of quantitative and qualitative analyses across different memory strategies, we show that, just like humans, carefully picking informative examples to be prioritized in cross-lingual memory replay helps tame the stability-plasticity dilemma. Compared to vanilla and strong memory replay baselines, our Leitner-guided approach significantly and consistently decreases forgetting while maintaining accuracy across natural language understanding tasks, language orders, and languages.
Anthology ID:
2024.naacl-long.432
Volume:
Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
7801–7814
Language:
URL:
https://aclanthology.org/2024.naacl-long.432
DOI:
Bibkey:
Cite (ACL):
Meryem M’hamdi and Jonathan May. 2024. Leitner-Guided Memory Replay for Cross-lingual Continual Learning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7801–7814, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
Leitner-Guided Memory Replay for Cross-lingual Continual Learning (M’hamdi & May, NAACL 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.naacl-long.432.pdf
Copyright:
 2024.naacl-long.432.copyright.pdf