Probing Multilingual BERT for Genetic and Typological Signals

Taraka Rama, Lisa Beinborn, Steffen Eger


Abstract
We probe the layers in multilingual BERT (mBERT) for phylogenetic and geographic language signals across 100 languages and compute language distances based on the mBERT representations. We 1) employ the language distances to infer and evaluate language trees, finding that they are close to the reference family tree in terms of quartet tree distance, 2) perform distance matrix regression analysis, finding that the language distances can be best explained by phylogenetic and worst by structural factors and 3) present a novel measure for measuring diachronic meaning stability (based on cross-lingual representation variability) which correlates significantly with published ranked lists based on linguistic approaches. Our results contribute to the nascent field of typological interpretability of cross-lingual text representations.
Anthology ID:
2020.coling-main.105
Volume:
Proceedings of the 28th International Conference on Computational Linguistics
Month:
December
Year:
2020
Address:
Barcelona, Spain (Online)
Editors:
Donia Scott, Nuria Bel, Chengqing Zong
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
1214–1228
Language:
URL:
https://aclanthology.org/2020.coling-main.105
DOI:
10.18653/v1/2020.coling-main.105
Bibkey:
Cite (ACL):
Taraka Rama, Lisa Beinborn, and Steffen Eger. 2020. Probing Multilingual BERT for Genetic and Typological Signals. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1214–1228, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Cite (Informal):
Probing Multilingual BERT for Genetic and Typological Signals (Rama et al., COLING 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.coling-main.105.pdf
Data
Panlex