Universal Dependencies According to BERT: Both More Specific and More General

Tomasz Limisiewicz, David Mareček, Rudolf Rosa


Abstract
This work focuses on analyzing the form and extent of syntactic abstraction captured by BERT by extracting labeled dependency trees from self-attentions. Previous work showed that individual BERT heads tend to encode particular dependency relation types. We extend these findings by explicitly comparing BERT relations to Universal Dependencies (UD) annotations, showing that they often do not match one-to-one. We suggest a method for relation identification and syntactic tree construction. Our approach produces significantly more consistent dependency trees than previous work, showing that it better explains the syntactic abstractions in BERT. At the same time, it can be successfully applied with only a minimal amount of supervision and generalizes well across languages.
Anthology ID:
2020.findings-emnlp.245
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2020
Month:
November
Year:
2020
Address:
Online
Venues:
EMNLP | Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2710–2722
Language:
URL:
https://aclanthology.org/2020.findings-emnlp.245
DOI:
10.18653/v1/2020.findings-emnlp.245
Bibkey:
Copy Citation:
PDF:
https://aclanthology.org/2020.findings-emnlp.245.pdf