Validity Assessment of Legal Will Statements as Natural Language Inference

Alice Kwak, Jacob Israelsen, Clayton Morrison, Derek Bambauer, Mihai Surdeanu


Abstract
This work introduces a natural language inference (NLI) dataset that focuses on the validity of statements in legal wills. This dataset is unique because: (a) each entailment decision requires three inputs: the statement from the will, the law, and the conditions that hold at the time of the testator’s death; and (b) the included texts are longer than the ones in current NLI datasets. We trained eight neural NLI models in this dataset. All the models achieve more than 80% macro F1 and accuracy, which indicates that neural approaches can handle this task reasonably well. However, group accuracy, a stricter evaluation measure that is calculated with a group of positive and negative examples generated from the same statement as a unit, is in mid 80s at best, which suggests that the models’ understanding of the task remains superficial. Further ablative analyses and explanation experiments indicate that all three text segments are used for prediction, but some decisions rely on semantically irrelevant tokens. This indicates that overfitting on these longer texts likely happens, and that additional research is required for this task to be solved.
Anthology ID:
2022.findings-emnlp.447
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2022
Month:
December
Year:
2022
Address:
Abu Dhabi, United Arab Emirates
Editors:
Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
6047–6056
Language:
URL:
https://aclanthology.org/2022.findings-emnlp.447
DOI:
10.18653/v1/2022.findings-emnlp.447
Bibkey:
Cite (ACL):
Alice Kwak, Jacob Israelsen, Clayton Morrison, Derek Bambauer, and Mihai Surdeanu. 2022. Validity Assessment of Legal Will Statements as Natural Language Inference. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6047–6056, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Cite (Informal):
Validity Assessment of Legal Will Statements as Natural Language Inference (Kwak et al., Findings 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.findings-emnlp.447.pdf
Video:
 https://aclanthology.org/2022.findings-emnlp.447.mp4