GENDER1PERSON: Test Suite for Estimating Gender Bias of First-person Singular Forms

Maja Popović, Ekaterina Lapshinova-Koltunski


Abstract
The gender1person test suite is designed to measure gender bias in translating singular first-person forms from English into two Slavic languages, Russian and Serbian. The test suite consists of 1,000 Amazon product reviews, uniformly distributed over 10 different product categories. Bias is measured through a gender score ranging from -100 (all reviews are feminine) to 100 (all reviews are masculine). The test suite shows that the majority of the systems participating in the WMT-2025 task for these two target languages prefer the masculine writer’s gender. There is no single system which is biased towards the feminine variant. Furthermore, for each language pair, there are seven systems that are considered balanced, having the gender scores between -10 and 10.Finally, the analysis of different products showed that the choice of the writer’s gender depends to a large extent on the product. Moreover, it is demonstrated that even the systems with overall balanced scores are actually biased, but in different ways for different product categories.
Anthology ID:
2025.wmt-1.56
Volume:
Proceedings of the Tenth Conference on Machine Translation
Month:
November
Year:
2025
Address:
Suzhou, China
Editors:
Barry Haddow, Tom Kocmi, Philipp Koehn, Christof Monz
Venue:
WMT
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
800–822
Language:
URL:
https://aclanthology.org/2025.wmt-1.56/
DOI:
Bibkey:
Cite (ACL):
Maja Popović and Ekaterina Lapshinova-Koltunski. 2025. GENDER1PERSON: Test Suite for Estimating Gender Bias of First-person Singular Forms. In Proceedings of the Tenth Conference on Machine Translation, pages 800–822, Suzhou, China. Association for Computational Linguistics.
Cite (Informal):
GENDER1PERSON: Test Suite for Estimating Gender Bias of First-person Singular Forms (Popović & Lapshinova-Koltunski, WMT 2025)
Copy Citation:
PDF:
https://aclanthology.org/2025.wmt-1.56.pdf