<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="bueno-etal-2024-mlissard">
    <titleInfo>
        <title>MLissard: Multilingual Long and Simple Sequential Reasoning Benchmarks</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Mirelle</namePart>
        <namePart type="given">Candida</namePart>
        <namePart type="family">Bueno</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Roberto</namePart>
        <namePart type="family">Lotufo</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Rodrigo</namePart>
        <namePart type="family">Frassetto Nogueira</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2nd GenBench Workshop on Generalisation (Benchmarking) in NLP</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Dieuwke</namePart>
            <namePart type="family">Hupkes</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Verna</namePart>
            <namePart type="family">Dankers</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Khuyagbaatar</namePart>
            <namePart type="family">Batsuren</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Amirhossein</namePart>
            <namePart type="family">Kazemnejad</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Christos</namePart>
            <namePart type="family">Christodoulopoulos</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Mario</namePart>
            <namePart type="family">Giulianelli</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ryan</namePart>
            <namePart type="family">Cotterell</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Miami, Florida, USA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Language models are now capable of solving tasks that require dealing with long sequences consisting of hundreds of thousands of tokens. However, they often fail on tasks that require repetitive use of simple rules, even on sequences that are much shorter than those seen during training. For example, state-of-the-art LLMs can find common items in two lists with up to 20 items but fail when lists have 80 items. In this paper, we introduce MLissard, a multilingual benchmark designed to evaluate models’ abilities to process and generate texts of varied lengths and offers a mechanism for controlling sequence complexity. Our evaluation of open-source and proprietary models show a consistent decline in performance across all models and languages as the complexity of the sequence increases. Surprisingly, the use of in-context examples in languages other than English helps increase extrapolation performance significantly.</abstract>
    <identifier type="citekey">bueno-etal-2024-mlissard</identifier>
    <location>
        <url>https://aclanthology.org/2024.genbench-1.6</url>
    </location>
    <part>
        <date>2024-11</date>
        <extent unit="page">
            <start>86</start>
            <end>95</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T MLissard: Multilingual Long and Simple Sequential Reasoning Benchmarks
%A Bueno, Mirelle Candida
%A Lotufo, Roberto
%A Frassetto Nogueira, Rodrigo
%Y Hupkes, Dieuwke
%Y Dankers, Verna
%Y Batsuren, Khuyagbaatar
%Y Kazemnejad, Amirhossein
%Y Christodoulopoulos, Christos
%Y Giulianelli, Mario
%Y Cotterell, Ryan
%S Proceedings of the 2nd GenBench Workshop on Generalisation (Benchmarking) in NLP
%D 2024
%8 November
%I Association for Computational Linguistics
%C Miami, Florida, USA
%F bueno-etal-2024-mlissard
%X Language models are now capable of solving tasks that require dealing with long sequences consisting of hundreds of thousands of tokens. However, they often fail on tasks that require repetitive use of simple rules, even on sequences that are much shorter than those seen during training. For example, state-of-the-art LLMs can find common items in two lists with up to 20 items but fail when lists have 80 items. In this paper, we introduce MLissard, a multilingual benchmark designed to evaluate models’ abilities to process and generate texts of varied lengths and offers a mechanism for controlling sequence complexity. Our evaluation of open-source and proprietary models show a consistent decline in performance across all models and languages as the complexity of the sequence increases. Surprisingly, the use of in-context examples in languages other than English helps increase extrapolation performance significantly.
%U https://aclanthology.org/2024.genbench-1.6
%P 86-95

Markdown (Informal)

[MLissard: Multilingual Long and Simple Sequential Reasoning Benchmarks](https://aclanthology.org/2024.genbench-1.6) (Bueno et al., GenBench 2024)

• MLissard: Multilingual Long and Simple Sequential Reasoning Benchmarks (Bueno et al., GenBench 2024)

ACL

• Mirelle Candida Bueno, Roberto Lotufo, and Rodrigo Frassetto Nogueira. 2024. MLissard: Multilingual Long and Simple Sequential Reasoning Benchmarks. In Proceedings of the 2nd GenBench Workshop on Generalisation (Benchmarking) in NLP, pages 86–95, Miami, Florida, USA. Association for Computational Linguistics.