Evaluating Gender Bias Transfer from Film Data

Amanda Bertsch, Ashley Oh, Sanika Natu, Swetha Gangu, Alan W. Black, Emma Strubell


Abstract
Films are a rich source of data for natural language processing. OpenSubtitles (Lison and Tiedemann, 2016) is a popular movie script dataset, used for training models for tasks such as machine translation and dialogue generation. However, movies often contain biases that reflect society at the time, and these biases may be introduced during pre-training and influence downstream models. We perform sentiment analysis on template infilling (Kurita et al., 2019) and the Sentence Embedding Association Test (May et al., 2019) to measure how BERT-based language models change after continued pre-training on OpenSubtitles. We consider gender bias as a primary motivating case for this analysis, while also measuring other social biases such as disability. We show that sentiment analysis on template infilling is not an effective measure of bias due to the rarity of disability and gender identifying tokens in the movie dialogue. We extend our analysis to a longitudinal study of bias in film dialogue over the last 110 years and find that continued pre-training on OpenSubtitles encodes additional bias into BERT. We show that BERT learns associations that reflect the biases and representation of each film era, suggesting that additional care must be taken when using historical data.
Anthology ID:
2022.gebnlp-1.24
Volume:
Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP)
Month:
July
Year:
2022
Address:
Seattle, Washington
Venue:
GeBNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
235–243
Language:
URL:
https://aclanthology.org/2022.gebnlp-1.24
DOI:
10.18653/v1/2022.gebnlp-1.24
Bibkey:
Cite (ACL):
Amanda Bertsch, Ashley Oh, Sanika Natu, Swetha Gangu, Alan W. Black, and Emma Strubell. 2022. Evaluating Gender Bias Transfer from Film Data. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 235–243, Seattle, Washington. Association for Computational Linguistics.
Cite (Informal):
Evaluating Gender Bias Transfer from Film Data (Bertsch et al., GeBNLP 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.gebnlp-1.24.pdf
Video:
 https://aclanthology.org/2022.gebnlp-1.24.mp4
Data
OpenSubtitles