<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="manjavacas-arevalo-fonteyn-2021-macberth">
    <titleInfo>
        <title>MacBERTh: Development and Evaluation of a Historically Pre-trained Language Model for English (1450-1950)</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Enrique</namePart>
        <namePart type="family">Manjavacas Arevalo</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Lauren</namePart>
        <namePart type="family">Fonteyn</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Workshop on Natural Language Processing for Digital Humanities</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Mika</namePart>
            <namePart type="family">Hämäläinen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Khalid</namePart>
            <namePart type="family">Alnajjar</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Niko</namePart>
            <namePart type="family">Partanen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jack</namePart>
            <namePart type="family">Rueter</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>NLP Association of India (NLPAI)</publisher>
            <place>
                <placeTerm type="text">NIT Silchar, India</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>The new pre-train-then-fine-tune paradigm in Natural made important performance gains accessible to a wider audience. Once pre-trained, deploying a large language model presents comparatively small infrastructure requirements, and offers robust performance in many NLP tasks. The Digital Humanities community has been an early adapter of this paradigm. Yet, a large part of this community is concerned with the application of NLP algorithms to historical texts, for which large models pre-trained on contemporary text may not provide optimal results. In the present paper, we present “MacBERTh”—a transformer-based language model pre-trained on historical English—and exhaustively assess its benefits on a large set of relevant downstream tasks. Our experiments highlight that, despite some differences across target time periods, pre-training on historical language from scratch outperforms models pre-trained on present-day language and later adapted to historical language.</abstract>
    <identifier type="citekey">manjavacas-arevalo-fonteyn-2021-macberth</identifier>
    <location>
        <url>https://aclanthology.org/2021.nlp4dh-1.4</url>
    </location>
    <part>
        <date>2021-12</date>
        <extent unit="page">
            <start>23</start>
            <end>36</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T MacBERTh: Development and Evaluation of a Historically Pre-trained Language Model for English (1450-1950)
%A Manjavacas Arevalo, Enrique
%A Fonteyn, Lauren
%Y Hämäläinen, Mika
%Y Alnajjar, Khalid
%Y Partanen, Niko
%Y Rueter, Jack
%S Proceedings of the Workshop on Natural Language Processing for Digital Humanities
%D 2021
%8 December
%I NLP Association of India (NLPAI)
%C NIT Silchar, India
%F manjavacas-arevalo-fonteyn-2021-macberth
%X The new pre-train-then-fine-tune paradigm in Natural made important performance gains accessible to a wider audience. Once pre-trained, deploying a large language model presents comparatively small infrastructure requirements, and offers robust performance in many NLP tasks. The Digital Humanities community has been an early adapter of this paradigm. Yet, a large part of this community is concerned with the application of NLP algorithms to historical texts, for which large models pre-trained on contemporary text may not provide optimal results. In the present paper, we present “MacBERTh”—a transformer-based language model pre-trained on historical English—and exhaustively assess its benefits on a large set of relevant downstream tasks. Our experiments highlight that, despite some differences across target time periods, pre-training on historical language from scratch outperforms models pre-trained on present-day language and later adapted to historical language.
%U https://aclanthology.org/2021.nlp4dh-1.4
%P 23-36

Markdown (Informal)

[MacBERTh: Development and Evaluation of a Historically Pre-trained Language Model for English (1450-1950)](https://aclanthology.org/2021.nlp4dh-1.4) (Manjavacas Arevalo & Fonteyn, NLP4DH 2021)

• MacBERTh: Development and Evaluation of a Historically Pre-trained Language Model for English (1450-1950) (Manjavacas Arevalo & Fonteyn, NLP4DH 2021)

ACL

• Enrique Manjavacas Arevalo and Lauren Fonteyn. 2021. MacBERTh: Development and Evaluation of a Historically Pre-trained Language Model for English (1450-1950). In Proceedings of the Workshop on Natural Language Processing for Digital Humanities, pages 23–36, NIT Silchar, India. NLP Association of India (NLPAI).