<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="kulmizev-etal-2020-neural">
    <titleInfo>
        <title>Do Neural Language Models Show Preferences for Syntactic Formalisms?</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Artur</namePart>
        <namePart type="family">Kulmizev</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Vinit</namePart>
        <namePart type="family">Ravishankar</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Mostafa</namePart>
        <namePart type="family">Abdou</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Joakim</namePart>
        <namePart type="family">Nivre</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Recent work on the interpretability of deep neural language models has concluded that many properties of natural language syntax are encoded in their representational spaces. However, such studies often suffer from limited scope by focusing on a single language and a single linguistic formalism. In this study, we aim to investigate the extent to which the semblance of syntactic structure captured by language models adheres to a surface-syntactic or deep syntactic style of analysis, and whether the patterns are consistent across different languages. We apply a probe for extracting directed dependency trees to BERT and ELMo models trained on 13 different languages, probing for two different syntactic annotation styles: Universal Dependencies (UD), prioritizing deep syntactic relations, and Surface-Syntactic Universal Dependencies (SUD), focusing on surface structure. We find that both models exhibit a preference for UD over SUD — with interesting variations across languages and layers — and that the strength of this preference is correlated with differences in tree shape.</abstract>
    <identifier type="citekey">kulmizev-etal-2020-neural</identifier>
    <identifier type="doi">10.18653/v1/2020.acl-main.375</identifier>
    <location>
        <url>https://aclanthology.org/2020.acl-main.375</url>
    </location>
    <part>
        <date>2020-07</date>
        <extent unit="page">
            <start>4077</start>
            <end>4091</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Do Neural Language Models Show Preferences for Syntactic Formalisms?
%A Kulmizev, Artur
%A Ravishankar, Vinit
%A Abdou, Mostafa
%A Nivre, Joakim
%S Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
%D 2020
%8 July
%I Association for Computational Linguistics
%C Online
%F kulmizev-etal-2020-neural
%X Recent work on the interpretability of deep neural language models has concluded that many properties of natural language syntax are encoded in their representational spaces. However, such studies often suffer from limited scope by focusing on a single language and a single linguistic formalism. In this study, we aim to investigate the extent to which the semblance of syntactic structure captured by language models adheres to a surface-syntactic or deep syntactic style of analysis, and whether the patterns are consistent across different languages. We apply a probe for extracting directed dependency trees to BERT and ELMo models trained on 13 different languages, probing for two different syntactic annotation styles: Universal Dependencies (UD), prioritizing deep syntactic relations, and Surface-Syntactic Universal Dependencies (SUD), focusing on surface structure. We find that both models exhibit a preference for UD over SUD — with interesting variations across languages and layers — and that the strength of this preference is correlated with differences in tree shape.
%R 10.18653/v1/2020.acl-main.375
%U https://aclanthology.org/2020.acl-main.375
%U https://doi.org/10.18653/v1/2020.acl-main.375
%P 4077-4091

Markdown (Informal)

[Do Neural Language Models Show Preferences for Syntactic Formalisms?](https://aclanthology.org/2020.acl-main.375) (Kulmizev et al., ACL 2020)

• Do Neural Language Models Show Preferences for Syntactic Formalisms? (Kulmizev et al., ACL 2020)

ACL

• Artur Kulmizev, Vinit Ravishankar, Mostafa Abdou, and Joakim Nivre. 2020. Do Neural Language Models Show Preferences for Syntactic Formalisms?. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4077–4091, Online. Association for Computational Linguistics.