<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="barale-etal-2023-language">
    <titleInfo>
        <title>Do Language Models Learn about Legal Entity Types during Pretraining?</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Claire</namePart>
        <namePart type="family">Barale</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Michael</namePart>
        <namePart type="family">Rovatsos</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Nehal</namePart>
        <namePart type="family">Bhuta</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Natural Legal Language Processing Workshop 2023</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Daniel</namePart>
            <namePart type="family">Preoțiuc-Pietro</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Catalina</namePart>
            <namePart type="family">Goanta</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ilias</namePart>
            <namePart type="family">Chalkidis</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Leslie</namePart>
            <namePart type="family">Barrett</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Gerasimos</namePart>
            <namePart type="family">Spanakis</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Nikolaos</namePart>
            <namePart type="family">Aletras</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Language Models (LMs) have proven their ability to acquire diverse linguistic knowledge during the pretraining phase, potentially serving as a valuable source of incidental supervision for downstream tasks. However, there has been limited research conducted on the retrieval of domain-specific knowledge, and specifically legal knowledge. We propose to explore the task of Entity Typing, serving as a proxy for evaluating legal knowledge as an essential aspect of text comprehension, and a foundational task to numerous downstream legal NLP applications. Through systematic evaluation and analysis and two types of prompting (cloze sentences and QA-based templates) and to clarify the nature of these acquired cues, we compare diverse types and lengths of entities both general and domain-specific entities, semantics or syntax signals, and different LM pretraining corpus (generic and legal-oriented) and architectures (encoder BERT-based and decoder-only with Llama2). We show that (1) Llama2 performs well on certain entities and exhibits potential for substantial improvement with optimized prompt templates, (2) law-oriented LMs show inconsistent performance, possibly due to variations in their training corpus, (3) LMs demonstrate the ability to type entities even in the case of multi-token entities, (4) all models struggle with entities belonging to sub-domains of the law (5) Llama2 appears to frequently overlook syntactic cues, a shortcoming less present in BERT-based architectures.</abstract>
    <identifier type="citekey">barale-etal-2023-language</identifier>
    <identifier type="doi">10.18653/v1/2023.nllp-1.4</identifier>
    <location>
        <url>https://aclanthology.org/2023.nllp-1.4/</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>25</start>
            <end>37</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Do Language Models Learn about Legal Entity Types during Pretraining?
%A Barale, Claire
%A Rovatsos, Michael
%A Bhuta, Nehal
%Y Preoțiuc-Pietro, Daniel
%Y Goanta, Catalina
%Y Chalkidis, Ilias
%Y Barrett, Leslie
%Y Spanakis, Gerasimos
%Y Aletras, Nikolaos
%S Proceedings of the Natural Legal Language Processing Workshop 2023
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F barale-etal-2023-language
%X Language Models (LMs) have proven their ability to acquire diverse linguistic knowledge during the pretraining phase, potentially serving as a valuable source of incidental supervision for downstream tasks. However, there has been limited research conducted on the retrieval of domain-specific knowledge, and specifically legal knowledge. We propose to explore the task of Entity Typing, serving as a proxy for evaluating legal knowledge as an essential aspect of text comprehension, and a foundational task to numerous downstream legal NLP applications. Through systematic evaluation and analysis and two types of prompting (cloze sentences and QA-based templates) and to clarify the nature of these acquired cues, we compare diverse types and lengths of entities both general and domain-specific entities, semantics or syntax signals, and different LM pretraining corpus (generic and legal-oriented) and architectures (encoder BERT-based and decoder-only with Llama2). We show that (1) Llama2 performs well on certain entities and exhibits potential for substantial improvement with optimized prompt templates, (2) law-oriented LMs show inconsistent performance, possibly due to variations in their training corpus, (3) LMs demonstrate the ability to type entities even in the case of multi-token entities, (4) all models struggle with entities belonging to sub-domains of the law (5) Llama2 appears to frequently overlook syntactic cues, a shortcoming less present in BERT-based architectures.
%R 10.18653/v1/2023.nllp-1.4
%U https://aclanthology.org/2023.nllp-1.4/
%U https://doi.org/10.18653/v1/2023.nllp-1.4
%P 25-37

Markdown (Informal)

[Do Language Models Learn about Legal Entity Types during Pretraining?](https://aclanthology.org/2023.nllp-1.4/) (Barale et al., NLLP 2023)

• Do Language Models Learn about Legal Entity Types during Pretraining? (Barale et al., NLLP 2023)

ACL

• Claire Barale, Michael Rovatsos, and Nehal Bhuta. 2023. Do Language Models Learn about Legal Entity Types during Pretraining?. In Proceedings of the Natural Legal Language Processing Workshop 2023, pages 25–37, Singapore. Association for Computational Linguistics.