<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="bhardwaj-etal-2023-adaptation">
    <titleInfo>
        <title>Adaptation Approaches for Nearest Neighbor Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Rishabh</namePart>
        <namePart type="family">Bhardwaj</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">George</namePart>
        <namePart type="family">Polovets</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Monica</namePart>
        <namePart type="family">Sunkara</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: ACL 2023</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Anna</namePart>
            <namePart type="family">Rogers</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jordan</namePart>
            <namePart type="family">Boyd-Graber</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Naoaki</namePart>
            <namePart type="family">Okazaki</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Toronto, Canada</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Semi-parametric Nearest Neighbor Language Models (kNN-LMs) have produced impressive gains over purely parametric LMs, by leveraging large-scale neighborhood retrieval over external memory datastores. However, there has been little investigation into adapting such models for new domains. This work attempts to fill that gap and suggests the following approaches for adapting kNN-LMs — 1) adapting the underlying LM (using Adapters), 2) expanding neighborhood retrieval over an additional adaptation datastore, and 3) adapting the weights (scores) of retrieved neighbors using a learned Rescorer module. We study each adaptation strategy separately, as well as the combined performance improvement through ablation experiments and an extensive set of evaluations run over seven adaptation domains. Our combined adaptation approach consistently outperforms purely parametric adaptation and zero-shot (kNN-LM) baselines that construct datastores from the adaptation data. On average, we see perplexity improvements of 17.1% and 16% for these respective baselines, across domains.</abstract>
    <identifier type="citekey">bhardwaj-etal-2023-adaptation</identifier>
    <identifier type="doi">10.18653/v1/2023.findings-acl.73</identifier>
    <location>
        <url>https://aclanthology.org/2023.findings-acl.73</url>
    </location>
    <part>
        <date>2023-07</date>
        <extent unit="page">
            <start>1135</start>
            <end>1146</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Adaptation Approaches for Nearest Neighbor Language Models
%A Bhardwaj, Rishabh
%A Polovets, George
%A Sunkara, Monica
%Y Rogers, Anna
%Y Boyd-Graber, Jordan
%Y Okazaki, Naoaki
%S Findings of the Association for Computational Linguistics: ACL 2023
%D 2023
%8 July
%I Association for Computational Linguistics
%C Toronto, Canada
%F bhardwaj-etal-2023-adaptation
%X Semi-parametric Nearest Neighbor Language Models (kNN-LMs) have produced impressive gains over purely parametric LMs, by leveraging large-scale neighborhood retrieval over external memory datastores. However, there has been little investigation into adapting such models for new domains. This work attempts to fill that gap and suggests the following approaches for adapting kNN-LMs — 1) adapting the underlying LM (using Adapters), 2) expanding neighborhood retrieval over an additional adaptation datastore, and 3) adapting the weights (scores) of retrieved neighbors using a learned Rescorer module. We study each adaptation strategy separately, as well as the combined performance improvement through ablation experiments and an extensive set of evaluations run over seven adaptation domains. Our combined adaptation approach consistently outperforms purely parametric adaptation and zero-shot (kNN-LM) baselines that construct datastores from the adaptation data. On average, we see perplexity improvements of 17.1% and 16% for these respective baselines, across domains.
%R 10.18653/v1/2023.findings-acl.73
%U https://aclanthology.org/2023.findings-acl.73
%U https://doi.org/10.18653/v1/2023.findings-acl.73
%P 1135-1146

Markdown (Informal)

[Adaptation Approaches for Nearest Neighbor Language Models](https://aclanthology.org/2023.findings-acl.73) (Bhardwaj et al., Findings 2023)

• Adaptation Approaches for Nearest Neighbor Language Models (Bhardwaj et al., Findings 2023)

ACL

• Rishabh Bhardwaj, George Polovets, and Monica Sunkara. 2023. Adaptation Approaches for Nearest Neighbor Language Models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1135–1146, Toronto, Canada. Association for Computational Linguistics.