Elastic weight consolidation for better bias inoculation

James Thorne, Andreas Vlachos


Abstract
The biases present in training datasets have been shown to affect models for sentence pair classification tasks such as natural language inference (NLI) and fact verification. While fine-tuning models on additional data has been used to mitigate them, a common issue is that of catastrophic forgetting of the original training dataset. In this paper, we show that elastic weight consolidation (EWC) allows fine-tuning of models to mitigate biases while being less susceptible to catastrophic forgetting. In our evaluation on fact verification and NLI stress tests, we show that fine-tuning with EWC dominates standard fine-tuning, yielding models with lower levels of forgetting on the original (biased) dataset for equivalent gains in accuracy on the fine-tuning (unbiased) dataset.
Anthology ID:
2021.eacl-main.82
Volume:
Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume
Month:
April
Year:
2021
Address:
Online
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
957–964
Language:
URL:
https://aclanthology.org/2021.eacl-main.82
DOI:
10.18653/v1/2021.eacl-main.82
Bibkey:
Cite (ACL):
James Thorne and Andreas Vlachos. 2021. Elastic weight consolidation for better bias inoculation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 957–964, Online. Association for Computational Linguistics.
Cite (Informal):
Elastic weight consolidation for better bias inoculation (Thorne & Vlachos, EACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.eacl-main.82.pdf
Code
 j6mes/eacl2021-debias-finetuning
Data
FEVERMultiFCMultiNLI