Aleyna Koçak

Author directory

2026

We introduce BenGER (Benchmark for German Law), a benchmark and dataset for subsumption-based legal reasoning in German law. It combines 596 exam-style free-text case tasks across several levels of legal education with 531 short doctrinal reasoning tasks, and a controlled subset adds timed human solutions written both unaided and in human-AI co-creation. We evaluate 12 contemporary LLM systems (closed flagship, efficiency-oriented, and open-weight) with a rubric-aligned LLM-as-a-Judge, which we cross-validate against a multi-rater human layer (three blind reviews per solution) and across six LLM judges. Our results show that closed-flagship systems lead across all three corpora, that human-AI co-creation measurably improves on unaided human work, and that the judge tracks human grading at Pearson r=0.76 and Cohen’s κ=0.60, with rankings stable across judges and two judges from independent providers clearing the Calderon single-reviewer replacement bar on human-authored solutions.