<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="saeed-papotti-2022-type">
    <titleInfo>
        <title>You Are My Type! Type Embeddings for Pre-trained Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Mohammed</namePart>
        <namePart type="family">Saeed</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Paolo</namePart>
        <namePart type="family">Papotti</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2022</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Yoav</namePart>
            <namePart type="family">Goldberg</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Zornitsa</namePart>
            <namePart type="family">Kozareva</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yue</namePart>
            <namePart type="family">Zhang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Abu Dhabi, United Arab Emirates</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>One reason for the positive impact of Pre-trained Language Models (PLMs) in NLP tasks is their ability to encode semantic types, such as ‘European City’ or ‘Woman’. While previous work has analyzed such information in the context of interpretability, it is not clear how to use types to steer the PLM output. For example, in a cloze statement, it is desirable to steer the model to generate a token that satisfies a user-specified type, e.g., predict a date rather than a location. In this work, we introduce Type Embeddings (TEs), an input embedding that promotes desired types in a PLM. Our proposal is to define a type by a small set of word examples. We empirically study the ability of TEs both in representing types and in steering masking predictions without changes to the prompt text in BERT. Finally, using the LAMA datasets, we show how TEs highly improve the precision in extracting facts from PLMs.</abstract>
    <identifier type="citekey">saeed-papotti-2022-type</identifier>
    <identifier type="doi">10.18653/v1/2022.findings-emnlp.336</identifier>
    <location>
        <url>https://aclanthology.org/2022.findings-emnlp.336</url>
    </location>
    <part>
        <date>2022-12</date>
        <extent unit="page">
            <start>4583</start>
            <end>4598</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T You Are My Type! Type Embeddings for Pre-trained Language Models
%A Saeed, Mohammed
%A Papotti, Paolo
%Y Goldberg, Yoav
%Y Kozareva, Zornitsa
%Y Zhang, Yue
%S Findings of the Association for Computational Linguistics: EMNLP 2022
%D 2022
%8 December
%I Association for Computational Linguistics
%C Abu Dhabi, United Arab Emirates
%F saeed-papotti-2022-type
%X One reason for the positive impact of Pre-trained Language Models (PLMs) in NLP tasks is their ability to encode semantic types, such as ‘European City’ or ‘Woman’. While previous work has analyzed such information in the context of interpretability, it is not clear how to use types to steer the PLM output. For example, in a cloze statement, it is desirable to steer the model to generate a token that satisfies a user-specified type, e.g., predict a date rather than a location. In this work, we introduce Type Embeddings (TEs), an input embedding that promotes desired types in a PLM. Our proposal is to define a type by a small set of word examples. We empirically study the ability of TEs both in representing types and in steering masking predictions without changes to the prompt text in BERT. Finally, using the LAMA datasets, we show how TEs highly improve the precision in extracting facts from PLMs.
%R 10.18653/v1/2022.findings-emnlp.336
%U https://aclanthology.org/2022.findings-emnlp.336
%U https://doi.org/10.18653/v1/2022.findings-emnlp.336
%P 4583-4598

Markdown (Informal)

[You Are My Type! Type Embeddings for Pre-trained Language Models](https://aclanthology.org/2022.findings-emnlp.336) (Saeed & Papotti, Findings 2022)

• You Are My Type! Type Embeddings for Pre-trained Language Models (Saeed & Papotti, Findings 2022)

ACL

• Mohammed Saeed and Paolo Papotti. 2022. You Are My Type! Type Embeddings for Pre-trained Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4583–4598, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.