Efficient Estimation of Influence of a Training Instance

Sosuke Kobayashi, Sho Yokoi, Jun Suzuki, Kentaro Inui


Abstract
Understanding the influence of a training instance on a neural network model leads to improving interpretability. However, it is difficult and inefficient to evaluate the influence, which shows how a model’s prediction would be changed if a training instance were not used. In this paper, we propose an efficient method for estimating the influence. Our method is inspired by dropout, which zero-masks a sub-network and prevents the sub-network from learning each training instance. By switching between dropout masks, we can use sub-networks that learned or did not learn each training instance and estimate its influence. Through experiments with BERT and VGGNet on classification datasets, we demonstrate that the proposed method can capture training influences, enhance the interpretability of error predictions, and cleanse the training dataset for improving generalization.
Anthology ID:
2020.sustainlp-1.6
Volume:
Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing
Month:
November
Year:
2020
Address:
Online
Venues:
EMNLP | sustainlp
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
41–47
Language:
URL:
https://aclanthology.org/2020.sustainlp-1.6
DOI:
10.18653/v1/2020.sustainlp-1.6
Bibkey:
Cite (ACL):
Sosuke Kobayashi, Sho Yokoi, Jun Suzuki, and Kentaro Inui. 2020. Efficient Estimation of Influence of a Training Instance. In Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, pages 41–47, Online. Association for Computational Linguistics.
Cite (Informal):
Efficient Estimation of Influence of a Training Instance (Kobayashi et al., sustainlp 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.sustainlp-1.6.pdf
Optional supplementary material:
 2020.sustainlp-1.6.OptionalSupplementaryMaterial.pdf
Video:
 https://slideslive.com/38939427
Data
CIFAR-10SST