How Do BERT Embeddings Organize Linguistic Knowledge?

Giovanni Puccetti, Alessio Miaschi, Felice Dell’Orletta


Abstract
Several studies investigated the linguistic information implicitly encoded in Neural Language Models. Most of these works focused on quantifying the amount and type of information available within their internal representations and across their layers. In line with this scenario, we proposed a different study, based on Lasso regression, aimed at understanding how the information encoded by BERT sentence-level representations is arrange within its hidden units. Using a suite of several probing tasks, we showed the existence of a relationship between the implicit knowledge learned by the model and the number of individual units involved in the encodings of this competence. Moreover, we found that it is possible to identify groups of hidden units more relevant for specific linguistic properties.
Anthology ID:
2021.deelio-1.6
Volume:
Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures
Month:
June
Year:
2021
Address:
Online
Venues:
DeeLIO | NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
48–57
Language:
URL:
https://aclanthology.org/2021.deelio-1.6
DOI:
10.18653/v1/2021.deelio-1.6
Bibkey:
Copy Citation:
PDF:
https://aclanthology.org/2021.deelio-1.6.pdf
Data
Universal Dependencies