<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="blair-stanek-etal-2024-blt">
    <titleInfo>
        <title>BLT: Can Large Language Models Handle Basic Legal Text?</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Andrew</namePart>
        <namePart type="family">Blair-Stanek</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Nils</namePart>
        <namePart type="family">Holzenberger</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Benjamin</namePart>
        <namePart type="family">Van Durme</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Natural Legal Language Processing Workshop 2024</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Nikolaos</namePart>
            <namePart type="family">Aletras</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ilias</namePart>
            <namePart type="family">Chalkidis</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Leslie</namePart>
            <namePart type="family">Barrett</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Cătălina</namePart>
            <namePart type="family">Goan\textcommabelowtă</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Daniel</namePart>
            <namePart type="family">Preo\textcommabelowtiuc-Pietro</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Gerasimos</namePart>
            <namePart type="family">Spanakis</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Miami, FL, USA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>We find that the best publicly available LLMs like GPT-4 and Claude currently perform poorly on basic legal text handling. This motivates the creation of a benchmark consisting of examples that lawyers and paralegals would expect LLMs to handle zero-shot, such as looking up the text at a line of a witness deposition or at a subsection of a contract. LLMs’ poor performance on this benchmark casts into doubt their reliability as-is for legal practice. However, fine-tuning on our training set brings even a small model to near-perfect performance. This benchmark will be useful for fine-tuning LLMs for downstream legal tasks, as well as for tracking LLMs’ reliability as-is for basic legal tasks.</abstract>
    <identifier type="citekey">blair-stanek-etal-2024-blt</identifier>
    <location>
        <url>https://aclanthology.org/2024.nllp-1.18</url>
    </location>
    <part>
        <date>2024-11</date>
        <extent unit="page">
            <start>216</start>
            <end>232</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T BLT: Can Large Language Models Handle Basic Legal Text?
%A Blair-Stanek, Andrew
%A Holzenberger, Nils
%A Van Durme, Benjamin
%Y Aletras, Nikolaos
%Y Chalkidis, Ilias
%Y Barrett, Leslie
%Y Goan\textcommabelowtă, Cătălina
%Y Preo\textcommabelowtiuc-Pietro, Daniel
%Y Spanakis, Gerasimos
%S Proceedings of the Natural Legal Language Processing Workshop 2024
%D 2024
%8 November
%I Association for Computational Linguistics
%C Miami, FL, USA
%F blair-stanek-etal-2024-blt
%X We find that the best publicly available LLMs like GPT-4 and Claude currently perform poorly on basic legal text handling. This motivates the creation of a benchmark consisting of examples that lawyers and paralegals would expect LLMs to handle zero-shot, such as looking up the text at a line of a witness deposition or at a subsection of a contract. LLMs’ poor performance on this benchmark casts into doubt their reliability as-is for legal practice. However, fine-tuning on our training set brings even a small model to near-perfect performance. This benchmark will be useful for fine-tuning LLMs for downstream legal tasks, as well as for tracking LLMs’ reliability as-is for basic legal tasks.
%U https://aclanthology.org/2024.nllp-1.18
%P 216-232

Markdown (Informal)

[BLT: Can Large Language Models Handle Basic Legal Text?](https://aclanthology.org/2024.nllp-1.18) (Blair-Stanek et al., NLLP 2024)

• BLT: Can Large Language Models Handle Basic Legal Text? (Blair-Stanek et al., NLLP 2024)

ACL

• Andrew Blair-Stanek, Nils Holzenberger, and Benjamin Van Durme. 2024. BLT: Can Large Language Models Handle Basic Legal Text?. In Proceedings of the Natural Legal Language Processing Workshop 2024, pages 216–232, Miami, FL, USA. Association for Computational Linguistics.