Multilingual BERT has an accent: Evaluating English influences on fluency in multilingual models

Isabel Papadimitriou, Kezia Lopez, Dan Jurafsky


Abstract
While multilingual language models can improve NLP performance on low-resource languages by leveraging higher-resource languages, they also reduce average performance on all languages (the ‘curse of multilinguality’). Here we show another problem with multilingual models: grammatical structures in higher-resource languages bleed into lower-resource languages, a phenomenon we call grammatical structure bias. We show this bias via a novel method for comparing the fluency of multilingual models to the fluency of monolingual Spanish and Greek models: testing their preference for two carefully-chosen variable grammatical structures (optional pronoun-drop in Spanish and optional Subject-Verb ordering in Greek). We find that multilingual BERT is biased toward the English-like setting (explicit pronouns and Subject-Verb-Object ordering) as compared to our monolingual control language model. With our case studies, we hope to bring to light the fine-grained ways in which multilingual models can be biased, and encourage more linguistically-aware fluency evaluation.
Anthology ID:
2023.findings-eacl.89
Volume:
Findings of the Association for Computational Linguistics: EACL 2023
Month:
May
Year:
2023
Address:
Dubrovnik, Croatia
Editors:
Andreas Vlachos, Isabelle Augenstein
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1194–1200
Language:
URL:
https://aclanthology.org/2023.findings-eacl.89
DOI:
10.18653/v1/2023.findings-eacl.89
Bibkey:
Cite (ACL):
Isabel Papadimitriou, Kezia Lopez, and Dan Jurafsky. 2023. Multilingual BERT has an accent: Evaluating English influences on fluency in multilingual models. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1194–1200, Dubrovnik, Croatia. Association for Computational Linguistics.
Cite (Informal):
Multilingual BERT has an accent: Evaluating English influences on fluency in multilingual models (Papadimitriou et al., Findings 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.findings-eacl.89.pdf
Video:
 https://aclanthology.org/2023.findings-eacl.89.mp4