Transferring Legal Natural Language Inference Model from a US State to Another: What Makes It So Hard?

Alice Kwak, Gaetano Forte, Derek Bambauer, Mihai Surdeanu


Abstract
This study investigates whether a legal natural language inference (NLI) model trained on the data from one US state can be transferred to another state. We fine-tuned a pre-trained model on the task of evaluating the validity of legal will statements, once with the dataset containing the Tennessee wills and once with the dataset containing the Idaho wills. Each model’s performance on the in-domain setting and the out-of-domain setting are compared to see if the models can across the states. We found that the model trained on one US state can be mostly transferred to another state. However, it is clear that the model’s performance drops in the out-of-domain setting. The F1 scores of the Tennessee model and the Idaho model are 96.41 and 92.03 when predicting the data from the same state, but they drop to 66.32 and 81.60 when predicting the data from another state. Subsequent error analysis revealed that there are two major sources of errors. First, the model fails to recognize equivalent laws across states when there are stylistic differences between laws. Second, difference in statutory section numbering system between the states makes it difficult for the model to locate laws relevant to the cases being predicted on. This analysis provides insights on how the future NLI system can be improved. Also, our findings offer empirical support to legal experts advocating the standardization of legal documents.
Anthology ID:
2023.nllp-1.21
Volume:
Proceedings of the Natural Legal Language Processing Workshop 2023
Month:
December
Year:
2023
Address:
Singapore
Editors:
Daniel Preoțiuc-Pietro, Catalina Goanta, Ilias Chalkidis, Leslie Barrett, Gerasimos Spanakis, Nikolaos Aletras
Venues:
NLLP | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
215–222
Language:
URL:
https://aclanthology.org/2023.nllp-1.21
DOI:
10.18653/v1/2023.nllp-1.21
Bibkey:
Cite (ACL):
Alice Kwak, Gaetano Forte, Derek Bambauer, and Mihai Surdeanu. 2023. Transferring Legal Natural Language Inference Model from a US State to Another: What Makes It So Hard?. In Proceedings of the Natural Legal Language Processing Workshop 2023, pages 215–222, Singapore. Association for Computational Linguistics.
Cite (Informal):
Transferring Legal Natural Language Inference Model from a US State to Another: What Makes It So Hard? (Kwak et al., NLLP-WS 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.nllp-1.21.pdf