<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="nair-modani-2023-exploiting">
    <titleInfo>
        <title>Exploiting Language Characteristics for Legal Domain-Specific Language Model Pretraining</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Inderjeet</namePart>
        <namePart type="family">Nair</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Natwar</namePart>
        <namePart type="family">Modani</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-05</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EACL 2023</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Andreas</namePart>
            <namePart type="family">Vlachos</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Isabelle</namePart>
            <namePart type="family">Augenstein</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Dubrovnik, Croatia</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Pretraining large language models has resulted in tremendous performance improvement for many natural language processing (NLP) tasks. While for non-domain specific tasks, such models can be used directly, a common strategy to achieve better performance for specific domains involves pretraining these language models over domain specific data using objectives like Masked Language Modelling (MLM), Autoregressive Language Modelling, etc. While such pretraining addresses the change in vocabulary and style of language for the domain, it is otherwise a domain agnostic approach. In this work, we investigate the effect of incorporating pretraining objectives that explicitly tries to exploit the domain specific language characteristics in addition to such MLM based pretraining. Particularly, we examine two distinct characteristics associated with the legal domain and propose pretraining objectives modelling these characteristics. The proposed objectives target improvement of token-level feature representation, as well as aim to incorporate sentence level semantics. We demonstrate superiority in the performance of the models pretrained using our objectives against those trained using domain-agnostic objectives over several legal downstream tasks.</abstract>
    <identifier type="citekey">nair-modani-2023-exploiting</identifier>
    <identifier type="doi">10.18653/v1/2023.findings-eacl.190</identifier>
    <location>
        <url>https://aclanthology.org/2023.findings-eacl.190</url>
    </location>
    <part>
        <date>2023-05</date>
        <extent unit="page">
            <start>2516</start>
            <end>2526</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Exploiting Language Characteristics for Legal Domain-Specific Language Model Pretraining
%A Nair, Inderjeet
%A Modani, Natwar
%Y Vlachos, Andreas
%Y Augenstein, Isabelle
%S Findings of the Association for Computational Linguistics: EACL 2023
%D 2023
%8 May
%I Association for Computational Linguistics
%C Dubrovnik, Croatia
%F nair-modani-2023-exploiting
%X Pretraining large language models has resulted in tremendous performance improvement for many natural language processing (NLP) tasks. While for non-domain specific tasks, such models can be used directly, a common strategy to achieve better performance for specific domains involves pretraining these language models over domain specific data using objectives like Masked Language Modelling (MLM), Autoregressive Language Modelling, etc. While such pretraining addresses the change in vocabulary and style of language for the domain, it is otherwise a domain agnostic approach. In this work, we investigate the effect of incorporating pretraining objectives that explicitly tries to exploit the domain specific language characteristics in addition to such MLM based pretraining. Particularly, we examine two distinct characteristics associated with the legal domain and propose pretraining objectives modelling these characteristics. The proposed objectives target improvement of token-level feature representation, as well as aim to incorporate sentence level semantics. We demonstrate superiority in the performance of the models pretrained using our objectives against those trained using domain-agnostic objectives over several legal downstream tasks.
%R 10.18653/v1/2023.findings-eacl.190
%U https://aclanthology.org/2023.findings-eacl.190
%U https://doi.org/10.18653/v1/2023.findings-eacl.190
%P 2516-2526

Markdown (Informal)

[Exploiting Language Characteristics for Legal Domain-Specific Language Model Pretraining](https://aclanthology.org/2023.findings-eacl.190) (Nair & Modani, Findings 2023)

• Exploiting Language Characteristics for Legal Domain-Specific Language Model Pretraining (Nair & Modani, Findings 2023)

ACL

• Inderjeet Nair and Natwar Modani. 2023. Exploiting Language Characteristics for Legal Domain-Specific Language Model Pretraining. In Findings of the Association for Computational Linguistics: EACL 2023, pages 2516–2526, Dubrovnik, Croatia. Association for Computational Linguistics.