Ann-Kristin Mayrhofer
Author directory2026
BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law
Sebastian Nagl | Ann-Kristin Mayrhofer | Martin Heidebach | Aleyna Koçak | Anne Zettelmeier | Elly Breu | Angelina Greiner | Sofija Milijas | Matthias Grabmair
Findings of the Association for Computational Linguistics: EMNLP 2026
Sebastian Nagl | Ann-Kristin Mayrhofer | Martin Heidebach | Aleyna Koçak | Anne Zettelmeier | Elly Breu | Angelina Greiner | Sofija Milijas | Matthias Grabmair
Findings of the Association for Computational Linguistics: EMNLP 2026
We introduce BenGER (Benchmark for German Law), a benchmark and dataset for subsumption-based legal reasoning in German law. It combines 596 exam-style free-text case tasks across several levels of legal education with 531 short doctrinal reasoning tasks, and a controlled subset adds timed human solutions written both unaided and in human-AI co-creation. We evaluate 12 contemporary LLM systems (closed flagship, efficiency-oriented, and open-weight) with a rubric-aligned LLM-as-a-Judge, which we cross-validate against a multi-rater human layer (three blind reviews per solution) and across six LLM judges. Our results show that closed-flagship systems lead across all three corpora, that human-AI co-creation measurably improves on unaided human work, and that the judge tracks human grading at Pearson r=0.76 and Cohen’s κ=0.60, with rankings stable across judges and two judges from independent providers clearing the Calderon single-reviewer replacement bar on human-authored solutions.