Team UTSA-NLP at SemEval 2024 Task 5: Prompt Ensembling for Argument Reasoning in Civil Procedures with GPT4

Dan Schumacher, Anthony Rios


Abstract
In this paper, we present our system for the SemEval Task 5, The Legal Argument Reasoning Task in Civil Procedure Challenge. Legal argument reasoning is an essential skill that all law students must master. Moreover, it is important to develop natural language processing solutions that can reason about a question given terse domain-specific contextual information. Our system explores a prompt-based solution using GPT4 to reason over legal arguments. We also evaluate an ensemble of prompting strategies, including chain-of-thought reasoning and in-context learning. Overall, our system results in a Macro F1 of .8095 on the validation dataset and .7315 (5th out of 21 teams) on the final test set. Code for this project is available at https://github.com/danschumac1/CivilPromptReasoningGPT4.
Anthology ID:
2024.semeval-1.187
Volume:
Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Atul Kr. Ojha, A. Seza Doğruöz, Harish Tayyar Madabushi, Giovanni Da San Martino, Sara Rosenthal, Aiala Rosá
Venue:
SemEval
SIG:
SIGLEX
Publisher:
Association for Computational Linguistics
Note:
Pages:
1293–1301
Language:
URL:
https://aclanthology.org/2024.semeval-1.187
DOI:
10.18653/v1/2024.semeval-1.187
Bibkey:
Cite (ACL):
Dan Schumacher and Anthony Rios. 2024. Team UTSA-NLP at SemEval 2024 Task 5: Prompt Ensembling for Argument Reasoning in Civil Procedures with GPT4. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 1293–1301, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
Team UTSA-NLP at SemEval 2024 Task 5: Prompt Ensembling for Argument Reasoning in Civil Procedures with GPT4 (Schumacher & Rios, SemEval 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.semeval-1.187.pdf
Supplementary material:
 2024.semeval-1.187.SupplementaryMaterial.zip
Supplementary material:
 2024.semeval-1.187.SupplementaryMaterial.txt