Mitigating the Diminishing Effect of Elastic Weight Consolidation

Canasai Kruengkrai, Junichi Yamagishi


Abstract
Elastic weight consolidation (EWC, Kirkpatrick et al. 2017) is a promising approach to addressing catastrophic forgetting in sequential training. We find that the effect of EWC can diminish when fine-tuning large-scale pre-trained language models on different datasets. We present two simple objective functions to mitigate this problem by rescaling the components of EWC. Experiments on natural language inference and fact-checking tasks indicate that our methods require much smaller values for the trade-off parameters to achieve results comparable to EWC.
Anthology ID:
2022.coling-1.403
Volume:
Proceedings of the 29th International Conference on Computational Linguistics
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea
Editors:
Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, Seung-Hoon Na
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
4568–4574
Language:
URL:
https://aclanthology.org/2022.coling-1.403
DOI:
Bibkey:
Cite (ACL):
Canasai Kruengkrai and Junichi Yamagishi. 2022. Mitigating the Diminishing Effect of Elastic Weight Consolidation. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4568–4574, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Cite (Informal):
Mitigating the Diminishing Effect of Elastic Weight Consolidation (Kruengkrai & Yamagishi, COLING 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.coling-1.403.pdf
Data
FEVERMultiNLI