<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="reinig-markert-2023-current">
    <titleInfo>
        <title>Can current NLI systems handle German word order? Investigating language model performance on a new German challenge set of minimal pairs</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Ines</namePart>
        <namePart type="family">Reinig</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Katja</namePart>
        <namePart type="family">Markert</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-06</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 15th International Conference on Computational Semantics</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Maxime</namePart>
            <namePart type="family">Amblard</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ellen</namePart>
            <namePart type="family">Breitholtz</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Nancy, France</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Compared to English, German word order is freer and therefore poses additional challenges for natural language inference (NLI). We create WOGLI (Word Order in German Language Inference), the first adversarial NLI dataset for German word order that has the following properties: (i) each premise has an entailed and a non-entailed hypothesis; (ii) premise and hypotheses differ only in word order and necessary morphological changes to mark case and number. In particular, each premise and its two hypotheses contain exactly the same lemmata. Our adversarial examples require the model to use morphological markers in order to recognise or reject entailment. We show that current German autoencoding models fine-tuned on translated NLI data can struggle on this challenge set, reflecting the fact that translated NLI datasets will not mirror all necessary language phenomena in the target language. We also examine performance after data augmentation as well as on related word order phenomena derived from WOGLI. Our datasets are publically available at https://github.com/ireinig/wogli.</abstract>
    <identifier type="citekey">reinig-markert-2023-current</identifier>
    <location>
        <url>https://aclanthology.org/2023.iwcs-1.1</url>
    </location>
    <part>
        <date>2023-06</date>
        <extent unit="page">
            <start>1</start>
            <end>15</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Can current NLI systems handle German word order? Investigating language model performance on a new German challenge set of minimal pairs
%A Reinig, Ines
%A Markert, Katja
%Y Amblard, Maxime
%Y Breitholtz, Ellen
%S Proceedings of the 15th International Conference on Computational Semantics
%D 2023
%8 June
%I Association for Computational Linguistics
%C Nancy, France
%F reinig-markert-2023-current
%X Compared to English, German word order is freer and therefore poses additional challenges for natural language inference (NLI). We create WOGLI (Word Order in German Language Inference), the first adversarial NLI dataset for German word order that has the following properties: (i) each premise has an entailed and a non-entailed hypothesis; (ii) premise and hypotheses differ only in word order and necessary morphological changes to mark case and number. In particular, each premise and its two hypotheses contain exactly the same lemmata. Our adversarial examples require the model to use morphological markers in order to recognise or reject entailment. We show that current German autoencoding models fine-tuned on translated NLI data can struggle on this challenge set, reflecting the fact that translated NLI datasets will not mirror all necessary language phenomena in the target language. We also examine performance after data augmentation as well as on related word order phenomena derived from WOGLI. Our datasets are publically available at https://github.com/ireinig/wogli.
%U https://aclanthology.org/2023.iwcs-1.1
%P 1-15

Markdown (Informal)

[Can current NLI systems handle German word order? Investigating language model performance on a new German challenge set of minimal pairs](https://aclanthology.org/2023.iwcs-1.1) (Reinig & Markert, IWCS 2023)

• Can current NLI systems handle German word order? Investigating language model performance on a new German challenge set of minimal pairs (Reinig & Markert, IWCS 2023)

ACL

• Ines Reinig and Katja Markert. 2023. Can current NLI systems handle German word order? Investigating language model performance on a new German challenge set of minimal pairs. In Proceedings of the 15th International Conference on Computational Semantics, pages 1–15, Nancy, France. Association for Computational Linguistics.