<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="agarwal-nenkova-2022-temporal">
    <titleInfo>
        <title>Temporal Effects on Pre-trained Models for Language Processing Tasks</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Oshin</namePart>
        <namePart type="family">Agarwal</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ani</namePart>
        <namePart type="family">Nenkova</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Keeping the performance of language technologies optimal as time passes is of great practical interest. We study temporal effects on model performance on downstream language tasks, establishing a nuanced terminology for such discussion and identifying factors essential to conduct a robust study. We present experiments for several tasks in English where the label correctness is not dependent on time and demonstrate the importance of distinguishing between temporal model deterioration and temporal domain adaptation for systems using pre-trained representations. We find that, depending on the task, temporal model deterioration is not necessarily a concern. Temporal domain adaptation, however, is beneficial in all cases, with better performance for a given time period possible when the system is trained on temporally more recent data. Therefore, we also examine the efficacy of two approaches for temporal domain adaptation without human annotations on new data. Self-labeling shows consistent improvement and notably, for named entity recognition, leads to better temporal adaptation than even human annotations.</abstract>
    <identifier type="citekey">agarwal-nenkova-2022-temporal</identifier>
    <identifier type="doi">10.1162/tacl_a_00497</identifier>
    <location>
        <url>https://aclanthology.org/2022.tacl-1.53</url>
    </location>
    <part>
        <date>2022</date>
        <detail type="volume"><number>10</number></detail>
        <extent unit="page">
            <start>904</start>
            <end>921</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T Temporal Effects on Pre-trained Models for Language Processing Tasks
%A Agarwal, Oshin
%A Nenkova, Ani
%J Transactions of the Association for Computational Linguistics
%D 2022
%V 10
%I MIT Press
%C Cambridge, MA
%F agarwal-nenkova-2022-temporal
%X Keeping the performance of language technologies optimal as time passes is of great practical interest. We study temporal effects on model performance on downstream language tasks, establishing a nuanced terminology for such discussion and identifying factors essential to conduct a robust study. We present experiments for several tasks in English where the label correctness is not dependent on time and demonstrate the importance of distinguishing between temporal model deterioration and temporal domain adaptation for systems using pre-trained representations. We find that, depending on the task, temporal model deterioration is not necessarily a concern. Temporal domain adaptation, however, is beneficial in all cases, with better performance for a given time period possible when the system is trained on temporally more recent data. Therefore, we also examine the efficacy of two approaches for temporal domain adaptation without human annotations on new data. Self-labeling shows consistent improvement and notably, for named entity recognition, leads to better temporal adaptation than even human annotations.
%R 10.1162/tacl_a_00497
%U https://aclanthology.org/2022.tacl-1.53
%U https://doi.org/10.1162/tacl_a_00497
%P 904-921

Markdown (Informal)

[Temporal Effects on Pre-trained Models for Language Processing Tasks](https://aclanthology.org/2022.tacl-1.53) (Agarwal & Nenkova, TACL 2022)

• Temporal Effects on Pre-trained Models for Language Processing Tasks (Agarwal & Nenkova, TACL 2022)

ACL

• Oshin Agarwal and Ani Nenkova. 2022. Temporal Effects on Pre-trained Models for Language Processing Tasks. Transactions of the Association for Computational Linguistics, 10:904–921.