Exploring the Effects of Negation and Grammatical Tense on Bias Probes

Samia Touileb


Abstract
We investigate in this paper how correlations between occupations and gendered-pronouns can be affected and changed by adding negation in bias probes, or changing the grammatical tense of the verbs in the probes. We use a set of simple bias probes in Norwegian and English, and perform 16 different probing analysis, using four Norwegian and four English pre-trained language models. We show that adding negation to probes does not have a considerable effect on the correlations between gendered-pronouns and occupations, supporting other works on negation in language models. We also show that altering the grammatical tense of verbs in bias probes do have some interesting effects on models’ behaviours and correlations. We argue that we should take grammatical tense into account when choosing bias probes, and aggregating results across tenses might be a better representation of the existing correlations.
Anthology ID:
2022.aacl-short.53
Volume:
Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)
Month:
November
Year:
2022
Address:
Online only
Editors:
Yulan He, Heng Ji, Sujian Li, Yang Liu, Chua-Hui Chang
Venues:
AACL | IJCNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
423–429
Language:
URL:
https://aclanthology.org/2022.aacl-short.53
DOI:
Bibkey:
Cite (ACL):
Samia Touileb. 2022. Exploring the Effects of Negation and Grammatical Tense on Bias Probes. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 423–429, Online only. Association for Computational Linguistics.
Cite (Informal):
Exploring the Effects of Negation and Grammatical Tense on Bias Probes (Touileb, AACL-IJCNLP 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.aacl-short.53.pdf
Dataset:
 2022.aacl-short.53.Dataset.zip