A Statutory Article Retrieval Dataset in French

Antoine Louis, Gerasimos Spanakis


Abstract
Statutory article retrieval is the task of automatically retrieving law articles relevant to a legal question. While recent advances in natural language processing have sparked considerable interest in many legal tasks, statutory article retrieval remains primarily untouched due to the scarcity of large-scale and high-quality annotated datasets. To address this bottleneck, we introduce the Belgian Statutory Article Retrieval Dataset (BSARD), which consists of 1,100+ French native legal questions labeled by experienced jurists with relevant articles from a corpus of 22,600+ Belgian law articles. Using BSARD, we benchmark several state-of-the-art retrieval approaches, including lexical and dense architectures, both in zero-shot and supervised setups. We find that fine-tuned dense retrieval models significantly outperform other systems. Our best performing baseline achieves 74.8% R@100, which is promising for the feasibility of the task and indicates there is still room for improvement. By the specificity of the domain and addressed task, BSARD presents a unique challenge problem for future research on legal information retrieval. Our dataset and source code are publicly available.
Anthology ID:
2022.acl-long.468
Volume:
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
May
Year:
2022
Address:
Dublin, Ireland
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
6789–6803
Language:
URL:
https://aclanthology.org/2022.acl-long.468
DOI:
10.18653/v1/2022.acl-long.468
Bibkey:
Cite (ACL):
Antoine Louis and Gerasimos Spanakis. 2022. A Statutory Article Retrieval Dataset in French. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6789–6803, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
A Statutory Article Retrieval Dataset in French (Louis & Spanakis, ACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.acl-long.468.pdf
Software:
 2022.acl-long.468.software.zip
Code
 maastrichtlawtech/bsard
Data
BSARD