<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="beloucif-biemann-2021-probing-pre">
    <titleInfo>
        <title>Probing Pre-trained Language Models for Semantic Attributes and their Values</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Meriem</namePart>
        <namePart type="family">Beloucif</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Chris</namePart>
        <namePart type="family">Biemann</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2021</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Marie-Francine</namePart>
            <namePart type="family">Moens</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Xuanjing</namePart>
            <namePart type="family">Huang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Lucia</namePart>
            <namePart type="family">Specia</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Scott</namePart>
            <namePart type="given">Wen-tau</namePart>
            <namePart type="family">Yih</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Punta Cana, Dominican Republic</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Pretrained language models (PTLMs) yield state-of-the-art performance on many natural language processing tasks, including syntax, semantics and commonsense. In this paper, we focus on identifying to what extent do PTLMs capture semantic attributes and their values, e.g., the correlation between rich and high net worth. We use PTLMs to predict masked tokens using patterns and lists of items from Wikidata in order to verify how likely PTLMs encode semantic attributes along with their values. Such inferences based on semantics are intuitive for humans as part of our language understanding. Since PTLMs are trained on large amount of Wikipedia data we would assume that they can generate similar predictions, yet our findings reveal that PTLMs are still much worse than humans on this task. We show evidence and analysis explaining how to exploit our methodology to integrate better context and semantics into PTLMs using knowledge bases.</abstract>
    <identifier type="citekey">beloucif-biemann-2021-probing-pre</identifier>
    <identifier type="doi">10.18653/v1/2021.findings-emnlp.218</identifier>
    <location>
        <url>https://aclanthology.org/2021.findings-emnlp.218</url>
    </location>
    <part>
        <date>2021-11</date>
        <extent unit="page">
            <start>2554</start>
            <end>2559</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Probing Pre-trained Language Models for Semantic Attributes and their Values
%A Beloucif, Meriem
%A Biemann, Chris
%Y Moens, Marie-Francine
%Y Huang, Xuanjing
%Y Specia, Lucia
%Y Yih, Scott Wen-tau
%S Findings of the Association for Computational Linguistics: EMNLP 2021
%D 2021
%8 November
%I Association for Computational Linguistics
%C Punta Cana, Dominican Republic
%F beloucif-biemann-2021-probing-pre
%X Pretrained language models (PTLMs) yield state-of-the-art performance on many natural language processing tasks, including syntax, semantics and commonsense. In this paper, we focus on identifying to what extent do PTLMs capture semantic attributes and their values, e.g., the correlation between rich and high net worth. We use PTLMs to predict masked tokens using patterns and lists of items from Wikidata in order to verify how likely PTLMs encode semantic attributes along with their values. Such inferences based on semantics are intuitive for humans as part of our language understanding. Since PTLMs are trained on large amount of Wikipedia data we would assume that they can generate similar predictions, yet our findings reveal that PTLMs are still much worse than humans on this task. We show evidence and analysis explaining how to exploit our methodology to integrate better context and semantics into PTLMs using knowledge bases.
%R 10.18653/v1/2021.findings-emnlp.218
%U https://aclanthology.org/2021.findings-emnlp.218
%U https://doi.org/10.18653/v1/2021.findings-emnlp.218
%P 2554-2559

Markdown (Informal)

[Probing Pre-trained Language Models for Semantic Attributes and their Values](https://aclanthology.org/2021.findings-emnlp.218) (Beloucif & Biemann, Findings 2021)

• Probing Pre-trained Language Models for Semantic Attributes and their Values (Beloucif & Biemann, Findings 2021)

ACL

• Meriem Beloucif and Chris Biemann. 2021. Probing Pre-trained Language Models for Semantic Attributes and their Values. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2554–2559, Punta Cana, Dominican Republic. Association for Computational Linguistics.