<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="kassner-etal-2020-pretrained">
    <titleInfo>
        <title>Are Pretrained Language Models Symbolic Reasoners over Knowledge?</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Nora</namePart>
        <namePart type="family">Kassner</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Benno</namePart>
        <namePart type="family">Krojer</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Hinrich</namePart>
        <namePart type="family">Schütze</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 24th Conference on Computational Natural Language Learning</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Raquel</namePart>
            <namePart type="family">Fernández</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Tal</namePart>
            <namePart type="family">Linzen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>How can pretrained language models (PLMs) learn factual knowledge from the training set? We investigate the two most important mechanisms: reasoning and memorization. Prior work has attempted to quantify the number of facts PLMs learn, but we present, using synthetic data, the first study that investigates the causal relation between facts present in training and facts learned by the PLM. For reasoning, we show that PLMs seem to learn to apply some symbolic reasoning rules correctly but struggle with others, including two-hop reasoning. Further analysis suggests that even the application of learned reasoning rules is flawed. For memorization, we identify schema conformity (facts systematically supported by other facts) and frequency as key factors for its success.</abstract>
    <identifier type="citekey">kassner-etal-2020-pretrained</identifier>
    <identifier type="doi">10.18653/v1/2020.conll-1.45</identifier>
    <location>
        <url>https://aclanthology.org/2020.conll-1.45</url>
    </location>
    <part>
        <date>2020-11</date>
        <extent unit="page">
            <start>552</start>
            <end>564</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Are Pretrained Language Models Symbolic Reasoners over Knowledge?
%A Kassner, Nora
%A Krojer, Benno
%A Schütze, Hinrich
%Y Fernández, Raquel
%Y Linzen, Tal
%S Proceedings of the 24th Conference on Computational Natural Language Learning
%D 2020
%8 November
%I Association for Computational Linguistics
%C Online
%F kassner-etal-2020-pretrained
%X How can pretrained language models (PLMs) learn factual knowledge from the training set? We investigate the two most important mechanisms: reasoning and memorization. Prior work has attempted to quantify the number of facts PLMs learn, but we present, using synthetic data, the first study that investigates the causal relation between facts present in training and facts learned by the PLM. For reasoning, we show that PLMs seem to learn to apply some symbolic reasoning rules correctly but struggle with others, including two-hop reasoning. Further analysis suggests that even the application of learned reasoning rules is flawed. For memorization, we identify schema conformity (facts systematically supported by other facts) and frequency as key factors for its success.
%R 10.18653/v1/2020.conll-1.45
%U https://aclanthology.org/2020.conll-1.45
%U https://doi.org/10.18653/v1/2020.conll-1.45
%P 552-564

Markdown (Informal)

[Are Pretrained Language Models Symbolic Reasoners over Knowledge?](https://aclanthology.org/2020.conll-1.45) (Kassner et al., CoNLL 2020)

• Are Pretrained Language Models Symbolic Reasoners over Knowledge? (Kassner et al., CoNLL 2020)

ACL

• Nora Kassner, Benno Krojer, and Hinrich Schütze. 2020. Are Pretrained Language Models Symbolic Reasoners over Knowledge?. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 552–564, Online. Association for Computational Linguistics.