<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="al-negheimish-etal-2023-towards">
    <titleInfo>
        <title>Towards preserving word order importance through Forced Invalidation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Hadeel</namePart>
        <namePart type="family">Al-Negheimish</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Pranava</namePart>
        <namePart type="family">Madhyastha</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Alessandra</namePart>
        <namePart type="family">Russo</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-05</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Andreas</namePart>
            <namePart type="family">Vlachos</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Isabelle</namePart>
            <namePart type="family">Augenstein</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Dubrovnik, Croatia</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Large pre-trained language models such as BERT have been widely used as a framework for natural language understanding (NLU) tasks. However, recent findings have revealed that pre-trained language models are insensitive to word order. The performance on NLU tasks remains unchanged even after randomly permuting the word of a sentence, where crucial syntactic information is destroyed. To help preserve the importance of word order, we propose a simple approach called Forced Invalidation (FI): forcing the model to identify permuted sequences as invalid samples. We perform an extensive evaluation of our approach on various English NLU and QA based tasks over BERT-based and attention-based models over word embeddings. Our experiments demonstrate that FI significantly improves the sensitivity of the models to word order.</abstract>
    <identifier type="citekey">al-negheimish-etal-2023-towards</identifier>
    <identifier type="doi">10.18653/v1/2023.eacl-main.187</identifier>
    <location>
        <url>https://aclanthology.org/2023.eacl-main.187</url>
    </location>
    <part>
        <date>2023-05</date>
        <extent unit="page">
            <start>2563</start>
            <end>2570</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Towards preserving word order importance through Forced Invalidation
%A Al-Negheimish, Hadeel
%A Madhyastha, Pranava
%A Russo, Alessandra
%Y Vlachos, Andreas
%Y Augenstein, Isabelle
%S Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics
%D 2023
%8 May
%I Association for Computational Linguistics
%C Dubrovnik, Croatia
%F al-negheimish-etal-2023-towards
%X Large pre-trained language models such as BERT have been widely used as a framework for natural language understanding (NLU) tasks. However, recent findings have revealed that pre-trained language models are insensitive to word order. The performance on NLU tasks remains unchanged even after randomly permuting the word of a sentence, where crucial syntactic information is destroyed. To help preserve the importance of word order, we propose a simple approach called Forced Invalidation (FI): forcing the model to identify permuted sequences as invalid samples. We perform an extensive evaluation of our approach on various English NLU and QA based tasks over BERT-based and attention-based models over word embeddings. Our experiments demonstrate that FI significantly improves the sensitivity of the models to word order.
%R 10.18653/v1/2023.eacl-main.187
%U https://aclanthology.org/2023.eacl-main.187
%U https://doi.org/10.18653/v1/2023.eacl-main.187
%P 2563-2570

Markdown (Informal)

[Towards preserving word order importance through Forced Invalidation](https://aclanthology.org/2023.eacl-main.187) (Al-Negheimish et al., EACL 2023)

• Towards preserving word order importance through Forced Invalidation (Al-Negheimish et al., EACL 2023)

ACL

• Hadeel Al-Negheimish, Pranava Madhyastha, and Alessandra Russo. 2023. Towards preserving word order importance through Forced Invalidation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2563–2570, Dubrovnik, Croatia. Association for Computational Linguistics.