<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="kwak-etal-2023-information">
    <titleInfo>
        <title>Information Extraction from Legal Wills: How Well Does GPT-4 Do?</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Alice</namePart>
        <namePart type="family">Kwak</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Cheonkam</namePart>
        <namePart type="family">Jeong</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Gaetano</namePart>
        <namePart type="family">Forte</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Derek</namePart>
        <namePart type="family">Bambauer</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Clayton</namePart>
        <namePart type="family">Morrison</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Mihai</namePart>
        <namePart type="family">Surdeanu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2023</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>This work presents a manually annotated dataset for Information Extraction (IE) from legal wills, and relevant in-context learning experiments on the dataset. The dataset consists of entities, binary relations between the entities (e.g., relations between testator and beneficiary), and n-ary events (e.g., bequest) extracted from 45 legal wills from two US states. This dataset can serve as a foundation for downstream tasks in the legal domain. Another use case of this dataset is evaluating the performance of large language models (LLMs) on this IE task. We evaluated GPT-4 with our dataset to investigate its ability to extract information from legal wills. Our evaluation result demonstrates that the model is capable of handling the task reasonably well. When given instructions and examples as a prompt, GPT-4 shows decent performance for both entity extraction and relation extraction tasks. Nevertheless, the evaluation result also reveals that the model is not perfect. We observed inconsistent outputs (given a prompt) as well as prompt over-generalization.</abstract>
    <identifier type="citekey">kwak-etal-2023-information</identifier>
    <identifier type="doi">10.18653/v1/2023.findings-emnlp.287</identifier>
    <location>
        <url>https://aclanthology.org/2023.findings-emnlp.287</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>4336</start>
            <end>4353</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Information Extraction from Legal Wills: How Well Does GPT-4 Do?
%A Kwak, Alice
%A Jeong, Cheonkam
%A Forte, Gaetano
%A Bambauer, Derek
%A Morrison, Clayton
%A Surdeanu, Mihai
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Findings of the Association for Computational Linguistics: EMNLP 2023
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F kwak-etal-2023-information
%X This work presents a manually annotated dataset for Information Extraction (IE) from legal wills, and relevant in-context learning experiments on the dataset. The dataset consists of entities, binary relations between the entities (e.g., relations between testator and beneficiary), and n-ary events (e.g., bequest) extracted from 45 legal wills from two US states. This dataset can serve as a foundation for downstream tasks in the legal domain. Another use case of this dataset is evaluating the performance of large language models (LLMs) on this IE task. We evaluated GPT-4 with our dataset to investigate its ability to extract information from legal wills. Our evaluation result demonstrates that the model is capable of handling the task reasonably well. When given instructions and examples as a prompt, GPT-4 shows decent performance for both entity extraction and relation extraction tasks. Nevertheless, the evaluation result also reveals that the model is not perfect. We observed inconsistent outputs (given a prompt) as well as prompt over-generalization.
%R 10.18653/v1/2023.findings-emnlp.287
%U https://aclanthology.org/2023.findings-emnlp.287
%U https://doi.org/10.18653/v1/2023.findings-emnlp.287
%P 4336-4353

Markdown (Informal)

[Information Extraction from Legal Wills: How Well Does GPT-4 Do?](https://aclanthology.org/2023.findings-emnlp.287) (Kwak et al., Findings 2023)

• Information Extraction from Legal Wills: How Well Does GPT-4 Do? (Kwak et al., Findings 2023)

ACL

• Alice Kwak, Cheonkam Jeong, Gaetano Forte, Derek Bambauer, Clayton Morrison, and Mihai Surdeanu. 2023. Information Extraction from Legal Wills: How Well Does GPT-4 Do?. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4336–4353, Singapore. Association for Computational Linguistics.