Rethinking Self-Attention: Towards Interpretability in Neural Parsing

Khalil Mrini, Franck Dernoncourt, Quan Hung Tran, Trung Bui, Walter Chang, Ndapa Nakashole


Abstract
Attention mechanisms have improved the performance of NLP tasks while allowing models to remain explainable. Self-attention is currently widely used, however interpretability is difficult due to the numerous attention distributions. Recent work has shown that model representations can benefit from label-specific information, while facilitating interpretation of predictions. We introduce the Label Attention Layer: a new form of self-attention where attention heads represent labels. We test our novel layer by running constituency and dependency parsing experiments and show our new model obtains new state-of-the-art results for both tasks on both the Penn Treebank (PTB) and Chinese Treebank. Additionally, our model requires fewer self-attention layers compared to existing work. Finally, we find that the Label Attention heads learn relations between syntactic categories and show pathways to analyze errors.
Anthology ID:
2020.findings-emnlp.65
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2020
Month:
November
Year:
2020
Address:
Online
Venues:
EMNLP | Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
731–742
Language:
URL:
https://aclanthology.org/2020.findings-emnlp.65
DOI:
10.18653/v1/2020.findings-emnlp.65
Bibkey:
Copy Citation:
PDF:
https://aclanthology.org/2020.findings-emnlp.65.pdf
Code
 KhalilMrini/LAL-Parser
Data
Penn Treebank