Low Frequency Names Exhibit Bias and Overfitting in Contextualizing Language Models

Robert Wolfe, Aylin Caliskan


Abstract
We use a dataset of U.S. first names with labels based on predominant gender and racial group to examine the effect of training corpus frequency on tokenization, contextualization, similarity to initial representation, and bias in BERT, GPT-2, T5, and XLNet. We show that predominantly female and non-white names are less frequent in the training corpora of these four language models. We find that infrequent names are more self-similar across contexts, with Spearman’s rho between frequency and self-similarity as low as -.763. Infrequent names are also less similar to initial representation, with Spearman’s rho between frequency and linear centered kernel alignment (CKA) similarity to initial representation as high as .702. Moreover, we find Spearman’s rho between racial bias and name frequency in BERT of .492, indicating that lower-frequency minority group names are more associated with unpleasantness. Representations of infrequent names undergo more processing, but are more self-similar, indicating that models rely on less context-informed representations of uncommon and minority names which are overfit to a lower number of observed contexts.
Anthology ID:
2021.emnlp-main.41
Erratum e1:
2021.emnlp-main.41e1
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Editors:
Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
518–532
Language:
URL:
https://aclanthology.org/2021.emnlp-main.41
DOI:
10.18653/v1/2021.emnlp-main.41
Bibkey:
Cite (ACL):
Robert Wolfe and Aylin Caliskan. 2021. Low Frequency Names Exhibit Bias and Overfitting in Contextualizing Language Models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 518–532, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Low Frequency Names Exhibit Bias and Overfitting in Contextualizing Language Models (Wolfe & Caliskan, EMNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.emnlp-main.41.pdf
Software:
 2021.emnlp-main.41.Software.zip
Video:
 https://aclanthology.org/2021.emnlp-main.41.mp4
Data
BookCorpusOpenWebTextWebText