Why Attention is Not Explanation: Surgical Intervention and Causal Reasoning about Neural Models

Christopher Grimsley, Elijah Mayfield, Julia R.S. Bursten


Abstract
As the demand for explainable deep learning grows in the evaluation of language technologies, the value of a principled grounding for those explanations grows as well. Here we study the state-of-the-art in explanation for neural models for NLP tasks from the viewpoint of philosophy of science. We focus on recent evaluation work that finds brittleness in explanations obtained through attention mechanisms. We harness philosophical accounts of explanation to suggest broader conclusions from these studies. From this analysis, we assert the impossibility of causal explanations from attention layers over text data. We then introduce NLP researchers to contemporary philosophy of science theories that allow robust yet non-causal reasoning in explanation, giving computer scientists a vocabulary for future research.
Anthology ID:
2020.lrec-1.220
Volume:
Proceedings of the Twelfth Language Resources and Evaluation Conference
Month:
May
Year:
2020
Address:
Marseille, France
Editors:
Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
1780–1790
Language:
English
URL:
https://aclanthology.org/2020.lrec-1.220
DOI:
Bibkey:
Cite (ACL):
Christopher Grimsley, Elijah Mayfield, and Julia R.S. Bursten. 2020. Why Attention is Not Explanation: Surgical Intervention and Causal Reasoning about Neural Models. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1780–1790, Marseille, France. European Language Resources Association.
Cite (Informal):
Why Attention is Not Explanation: Surgical Intervention and Causal Reasoning about Neural Models (Grimsley et al., LREC 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.lrec-1.220.pdf