Leon Schreiber
Author directory2026
DEval-MT: An Intersectional Gender Bias Evaluation Framework for German Machine Translation
Michelle Kappl | Leon Schreiber | Stefan Hillmann
Proceedings of the 7th Workshop on Gender Bias in Natural Language Processing (GeBNLP)
Michelle Kappl | Leon Schreiber | Stefan Hillmann
Proceedings of the 7th Workshop on Gender Bias in Natural Language Processing (GeBNLP)
Machine translation systems can reproduce gender bias, but most evaluation resources are limited to English as the source language. We introduce DEval-MT, a template-based framework for evaluating gender bias in German-source machine translation, comprising 6,618 sentences across six sentence styles with an intersectional template design that can be extended with additional demographic attributes and support for nonbinary neopronouns. Alongside it, we provide a Python package for translation, alignment, gender prediction, and analysis. Evaluating six different MT systems on seven target languages using this dataset and pipeline, we find evidence of representational harms across multiple demographic groups.