Generalizing Backpropagation for Gradient-Based Interpretability
Kevin
Du
author
Lucas
Torroba Hennigen
author
Niklas
Stoehr
author
Alex
Warstadt
author
Ryan
Cotterell
author
2023-07
text
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Anna
Rogers
editor
Jordan
Boyd-Graber
editor
Naoaki
Okazaki
editor
Association for Computational Linguistics
Toronto, Canada
conference publication
Many popular feature-attribution methods for interpreting deep neural networks rely on computing the gradients of a model’s output with respect to its inputs. While these methods can indicate which input features may be important for the model’s prediction, they reveal little about the inner workings of the model itself. In this paper, we observe that the gradient computation of a model is a special case of a more general formulation using semirings. This observation allows us to generalize the backpropagation algorithm to efficiently compute other interpretable statistics about the gradient graph of a neural network, such as the highest-weighted path and entropy. We implement this generalized algorithm, evaluate it on synthetic datasets to better understand the statistics it computes, and apply it to study BERT’s behavior on the subject–verb number agreement task (SVA). With this method, we (a) validate that the amount of gradient flow through a component of a model reflects its importance to a prediction and (b) for SVA, identify which pathways of the self-attention mechanism are most important.
du-etal-2023-generalizing
10.18653/v1/2023.acl-long.669
https://aclanthology.org/2023.acl-long.669
2023-07
11979
11995