How Does Stereotype Content Differ across Data Sources?

Kathleen Fraser, Svetlana Kiritchenko, Isar Nejadgholi


Abstract
For decades, psychologists have been studying stereotypes using specially-designed rating scales to capture people’s beliefs and opinions about different social groups. Now, using NLP tools on extensive collections of text, we have the opportunity to study stereotypes “in the wild” and on a large scale. However, are we truly capturing the same information? In this paper we compare measurements along six psychologically-motivated, stereotype-relevant dimensions (Sociability, Morality, Ability, Assertiveness, Beliefs, and Status) for 10 groups, defined by occupation. We compute these measurements on stereotypical English sentences written by crowd-workers, stereotypical sentences generated by ChatGPT, and more general data collected from social media, and contrast the findings with traditional, survey-based results, as well as a spontaneous word-list generation task. We find that while the correlation with the traditional scales varies across dimensions, the free-text data can be used to specify the particular traits associated with each group, and provide context for numerical survey data.
Anthology ID:
2024.starsem-1.2
Volume:
Proceedings of the 13th Joint Conference on Lexical and Computational Semantics (*SEM 2024)
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Danushka Bollegala, Vered Shwartz
Venue:
*SEM
SIG:
SIGLEX
Publisher:
Association for Computational Linguistics
Note:
Pages:
18–34
Language:
URL:
https://aclanthology.org/2024.starsem-1.2
DOI:
10.18653/v1/2024.starsem-1.2
Bibkey:
Cite (ACL):
Kathleen Fraser, Svetlana Kiritchenko, and Isar Nejadgholi. 2024. How Does Stereotype Content Differ across Data Sources?. In Proceedings of the 13th Joint Conference on Lexical and Computational Semantics (*SEM 2024), pages 18–34, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
How Does Stereotype Content Differ across Data Sources? (Fraser et al., *SEM 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.starsem-1.2.pdf