Sense Embeddings are also Biased – Evaluating Social Biases in Static and Contextualised Sense Embeddings

Yi Zhou, Masahiro Kaneko, Danushka Bollegala


Abstract
Sense embedding learning methods learn different embeddings for the different senses of an ambiguous word. One sense of an ambiguous word might be socially biased while its other senses remain unbiased. In comparison to the numerous prior work evaluating the social biases in pretrained word embeddings, the biases in sense embeddings have been relatively understudied. We create a benchmark dataset for evaluating the social biases in sense embeddings and propose novel sense-specific bias evaluation measures. We conduct an extensive evaluation of multiple static and contextualised sense embeddings for various types of social biases using the proposed measures. Our experimental results show that even in cases where no biases are found at word-level, there still exist worrying levels of social biases at sense-level, which are often ignored by the word-level bias evaluation measures.
Anthology ID:
2022.acl-long.135
Volume:
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Smaranda Muresan, Preslav Nakov, Aline Villavicencio
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1924–1935
Language:
URL:
https://aclanthology.org/2022.acl-long.135
DOI:
10.18653/v1/2022.acl-long.135
Bibkey:
Cite (ACL):
Yi Zhou, Masahiro Kaneko, and Danushka Bollegala. 2022. Sense Embeddings are also Biased – Evaluating Social Biases in Static and Contextualised Sense Embeddings. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1924–1935, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Sense Embeddings are also Biased – Evaluating Social Biases in Static and Contextualised Sense Embeddings (Zhou et al., ACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.acl-long.135.pdf
Software:
 2022.acl-long.135.software.zip
Video:
 https://aclanthology.org/2022.acl-long.135.mp4
Data
CrowS-PairsStereoSet