<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="hessel-schofield-2021-effective">
    <titleInfo>
        <title>How effective is BERT without word ordering? Implications for language understanding and data privacy</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Jack</namePart>
        <namePart type="family">Hessel</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Alexandra</namePart>
        <namePart type="family">Schofield</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-08</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Chengqing</namePart>
            <namePart type="family">Zong</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Fei</namePart>
            <namePart type="family">Xia</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Wenjie</namePart>
            <namePart type="family">Li</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Roberto</namePart>
            <namePart type="family">Navigli</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Ordered word sequences contain the rich structures that define language. However, it’s often not clear if or how modern pretrained language models utilize these structures. We show that the token representations and self-attention activations within BERT are surprisingly resilient to shuffling the order of input tokens, and that for several GLUE language understanding tasks, shuffling only minimally degrades performance, e.g., by 4% for QNLI. While bleak from the perspective of language understanding, our results have positive implications for cases where copyright or ethics necessitates the consideration of bag-of-words data (vs. full documents). We simulate such a scenario for three sensitive classification tasks, demonstrating minimal performance degradation vs. releasing full language sequences.</abstract>
    <identifier type="citekey">hessel-schofield-2021-effective</identifier>
    <identifier type="doi">10.18653/v1/2021.acl-short.27</identifier>
    <location>
        <url>https://aclanthology.org/2021.acl-short.27</url>
    </location>
    <part>
        <date>2021-08</date>
        <extent unit="page">
            <start>204</start>
            <end>211</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T How effective is BERT without word ordering? Implications for language understanding and data privacy
%A Hessel, Jack
%A Schofield, Alexandra
%Y Zong, Chengqing
%Y Xia, Fei
%Y Li, Wenjie
%Y Navigli, Roberto
%S Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)
%D 2021
%8 August
%I Association for Computational Linguistics
%C Online
%F hessel-schofield-2021-effective
%X Ordered word sequences contain the rich structures that define language. However, it’s often not clear if or how modern pretrained language models utilize these structures. We show that the token representations and self-attention activations within BERT are surprisingly resilient to shuffling the order of input tokens, and that for several GLUE language understanding tasks, shuffling only minimally degrades performance, e.g., by 4% for QNLI. While bleak from the perspective of language understanding, our results have positive implications for cases where copyright or ethics necessitates the consideration of bag-of-words data (vs. full documents). We simulate such a scenario for three sensitive classification tasks, demonstrating minimal performance degradation vs. releasing full language sequences.
%R 10.18653/v1/2021.acl-short.27
%U https://aclanthology.org/2021.acl-short.27
%U https://doi.org/10.18653/v1/2021.acl-short.27
%P 204-211

Markdown (Informal)

[How effective is BERT without word ordering? Implications for language understanding and data privacy](https://aclanthology.org/2021.acl-short.27) (Hessel & Schofield, ACL-IJCNLP 2021)

• How effective is BERT without word ordering? Implications for language understanding and data privacy (Hessel & Schofield, ACL-IJCNLP 2021)

ACL

• Jack Hessel and Alexandra Schofield. 2021. How effective is BERT without word ordering? Implications for language understanding and data privacy. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 204–211, Online. Association for Computational Linguistics.