Low-Resource Deontic Modality Classification in EU Legislation

Kristina Minkova, Shashank Chakravarthy, Gijs Dijck


Abstract
In law, it is important to distinguish between obligations, permissions, prohibitions, rights, and powers. These categories are called deontic modalities. This paper evaluates the performance of two deontic modality classification models, LEGAL-BERT and a Fusion model, in a low-resource setting. To create a generalized dataset for multi-class classification, we extracted random provisions from European Union (EU) legislation. By fine-tuning previously researched and published models, we evaluate their performance on our dataset against fusion models designed for low-resource text classification. We incorporate focal loss as an alternative for cross-entropy to tackle issues of class imbalance. The experiments indicate that the fusion model performs better for both balanced and imbalanced data with a macro F1-score of 0.61 for imbalanced data, 0.62 for balanced data, and 0.55 with focal loss for imbalanced data. When focusing on accuracy, our experiments indicate that the fusion model performs better with scores of 0.91 for imbalanced data, 0.78 for balanced data, and 0.90 for imbalanced data with focal loss.
Anthology ID:
2023.nllp-1.15
Volume:
Proceedings of the Natural Legal Language Processing Workshop 2023
Month:
December
Year:
2023
Address:
Singapore
Editors:
Daniel Preoțiuc-Pietro, Catalina Goanta, Ilias Chalkidis, Leslie Barrett, Gerasimos (Jerry) Spanakis, Nikolaos Aletras
Venues:
NLLP | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
149–158
Language:
URL:
https://aclanthology.org/2023.nllp-1.15
DOI:
10.18653/v1/2023.nllp-1.15
Bibkey:
Cite (ACL):
Kristina Minkova, Shashank Chakravarthy, and Gijs Dijck. 2023. Low-Resource Deontic Modality Classification in EU Legislation. In Proceedings of the Natural Legal Language Processing Workshop 2023, pages 149–158, Singapore. Association for Computational Linguistics.
Cite (Informal):
Low-Resource Deontic Modality Classification in EU Legislation (Minkova et al., NLLP-WS 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.nllp-1.15.pdf