FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing

Ilias Chalkidis, Tommaso Pasini, Sheng Zhang, Letizia Tomada, Sebastian Schwemer, Anders Søgaard


Abstract
We present a benchmark suite of four datasets for evaluating the fairness of pre-trained language models and the techniques used to fine-tune them for downstream tasks. Our benchmarks cover four jurisdictions (European Council, USA, Switzerland, and China), five languages (English, German, French, Italian and Chinese) and fairness across five attributes (gender, age, region, language, and legal area). In our experiments, we evaluate pre-trained language models using several group-robust fine-tuning techniques and show that performance group disparities are vibrant in many cases, while none of these techniques guarantee fairness, nor consistently mitigate group disparities. Furthermore, we provide a quantitative and qualitative analysis of our results, highlighting open challenges in the development of robustness methods in legal NLP.
Anthology ID:
2022.acl-long.301
Volume:
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Smaranda Muresan, Preslav Nakov, Aline Villavicencio
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4389–4406
Language:
URL:
https://aclanthology.org/2022.acl-long.301
DOI:
10.18653/v1/2022.acl-long.301
Bibkey:
Cite (ACL):
Ilias Chalkidis, Tommaso Pasini, Sheng Zhang, Letizia Tomada, Sebastian Schwemer, and Anders Søgaard. 2022. FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4389–4406, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing (Chalkidis et al., ACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.acl-long.301.pdf
Software:
 2022.acl-long.301.software.zip
Video:
 https://aclanthology.org/2022.acl-long.301.mp4
Code
 coastalcph/fairlex
Data
ECtHR