Mitigating the Diminishing Effect of Elastic Weight Consolidation

Canasai Kruengkrai, Junichi Yamagishi


Abstract
Elastic weight consolidation (EWC, Kirkpatrick et al. 2017) is a promising approach to addressing catastrophic forgetting in sequential training. We find that the effect of EWC can diminish when fine-tuning large-scale pre-trained language models on different datasets. We present two simple objective functions to mitigate this problem by rescaling the components of EWC. Experiments on natural language inference and fact-checking tasks indicate that our methods require much smaller values for the trade-off parameters to achieve results comparable to EWC.
Anthology ID:
2022.coling-1.403
Volume:
Proceedings of the 29th International Conference on Computational Linguistics
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
4568–4574
Language:
URL:
https://aclanthology.org/2022.coling-1.403
DOI:
Bibkey:
Cite (ACL):
Canasai Kruengkrai and Junichi Yamagishi. 2022. Mitigating the Diminishing Effect of Elastic Weight Consolidation. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4568–4574, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Cite (Informal):
Mitigating the Diminishing Effect of Elastic Weight Consolidation (Kruengkrai & Yamagishi, COLING 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.coling-1.403.pdf
Data
FEVERMultiNLI