Leon Schreiber

Author directory

2026

Machine translation systems can reproduce gender bias, but most evaluation resources are limited to English as the source language. We introduce DEval-MT, a template-based framework for evaluating gender bias in German-source machine translation, comprising 6,618 sentences across six sentence styles with an intersectional template design that can be extended with additional demographic attributes and support for nonbinary neopronouns. Alongside it, we provide a Python package for translation, alignment, gender prediction, and analysis. Evaluating six different MT systems on seven target languages using this dataset and pipeline, we find evidence of representational harms across multiple demographic groups.