<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="schick-schutze-2020-bertram">
    <titleInfo>
        <title>BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performance</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Timo</namePart>
        <namePart type="family">Schick</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Hinrich</namePart>
        <namePart type="family">Schütze</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Dan</namePart>
            <namePart type="family">Jurafsky</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Joyce</namePart>
            <namePart type="family">Chai</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Natalie</namePart>
            <namePart type="family">Schluter</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Joel</namePart>
            <namePart type="family">Tetreault</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Pretraining deep language models has led to large performance gains in NLP. Despite this success, Schick and Schütze (2020) recently showed that these models struggle to understand rare words. For static word embeddings, this problem has been addressed by separately learning representations for rare words. In this work, we transfer this idea to pretrained language models: We introduce BERTRAM, a powerful architecture based on BERT that is capable of inferring high-quality embeddings for rare words that are suitable as input representations for deep language models. This is achieved by enabling the surface form and contexts of a word to interact with each other in a deep architecture. Integrating BERTRAM into BERT leads to large performance increases due to improved representations of rare and medium frequency words on both a rare word probing task and three downstream tasks.</abstract>
    <identifier type="citekey">schick-schutze-2020-bertram</identifier>
    <identifier type="doi">10.18653/v1/2020.acl-main.368</identifier>
    <location>
        <url>https://aclanthology.org/2020.acl-main.368</url>
    </location>
    <part>
        <date>2020-07</date>
        <extent unit="page">
            <start>3996</start>
            <end>4007</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performance
%A Schick, Timo
%A Schütze, Hinrich
%Y Jurafsky, Dan
%Y Chai, Joyce
%Y Schluter, Natalie
%Y Tetreault, Joel
%S Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
%D 2020
%8 July
%I Association for Computational Linguistics
%C Online
%F schick-schutze-2020-bertram
%X Pretraining deep language models has led to large performance gains in NLP. Despite this success, Schick and Schütze (2020) recently showed that these models struggle to understand rare words. For static word embeddings, this problem has been addressed by separately learning representations for rare words. In this work, we transfer this idea to pretrained language models: We introduce BERTRAM, a powerful architecture based on BERT that is capable of inferring high-quality embeddings for rare words that are suitable as input representations for deep language models. This is achieved by enabling the surface form and contexts of a word to interact with each other in a deep architecture. Integrating BERTRAM into BERT leads to large performance increases due to improved representations of rare and medium frequency words on both a rare word probing task and three downstream tasks.
%R 10.18653/v1/2020.acl-main.368
%U https://aclanthology.org/2020.acl-main.368
%U https://doi.org/10.18653/v1/2020.acl-main.368
%P 3996-4007

Markdown (Informal)

[BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performance](https://aclanthology.org/2020.acl-main.368) (Schick & Schütze, ACL 2020)

• BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performance (Schick & Schütze, ACL 2020)

ACL

• Timo Schick and Hinrich Schütze. 2020. BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performance. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3996–4007, Online. Association for Computational Linguistics.