Questions about Contracts: Prompt Templates for Structured Answer Generation

Adam Roegiest, Radha Chitta, Jonathan Donnelly, Maya Lash, Alexandra Vtyurina, Francois Longtin


Abstract
Finding the answers to legal questions about specific clauses in contracts is an important analysis in many legal workflows (e.g., understanding market trends, due diligence, risk mitigation) but more important is being able to do this at scale. In this paper, we present an examination of using large language models to produce (partially) structured answers to legal questions; primarily in the form of multiple choice and multiple select. We first show that traditional semantic matching is unable to perform this task at acceptable accuracy and then show how question specific prompts can achieve reasonable accuracy across a range of generative models. Finally, we show that much of this effectiveness can be maintained when generalized prompt templates are used rather than question specific ones.
Anthology ID:
2023.nllp-1.8
Volume:
Proceedings of the Natural Legal Language Processing Workshop 2023
Month:
December
Year:
2023
Address:
Singapore
Editors:
Daniel Preoțiuc-Pietro, Catalina Goanta, Ilias Chalkidis, Leslie Barrett, Gerasimos Spanakis, Nikolaos Aletras
Venues:
NLLP | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
62–72
Language:
URL:
https://aclanthology.org/2023.nllp-1.8
DOI:
10.18653/v1/2023.nllp-1.8
Bibkey:
Cite (ACL):
Adam Roegiest, Radha Chitta, Jonathan Donnelly, Maya Lash, Alexandra Vtyurina, and Francois Longtin. 2023. Questions about Contracts: Prompt Templates for Structured Answer Generation. In Proceedings of the Natural Legal Language Processing Workshop 2023, pages 62–72, Singapore. Association for Computational Linguistics.
Cite (Informal):
Questions about Contracts: Prompt Templates for Structured Answer Generation (Roegiest et al., NLLP-WS 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.nllp-1.8.pdf