MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs

Jaap Jumelet; Leonie Weissweiler; Joakim Nivre; Arianna Bisazza

doi:10.1162/tacl.a.600

MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs

Jaap Jumelet, Leonie Weissweiler, Joakim Nivre, Arianna Bisazza

Correct Metadata for

Use this form to create a GitHub issue with structured data describing the correction. You will need a GitHub account. Once you create that issue, the correction will be reviewed by a staff member.

⚠️ Mobile Users: Submitting this form to create a new issue will only work with github.com, not the GitHub Mobile app.

Important: The Anthology treat PDFs as authoritative. Please use this form only to correct data that is out of line with the PDF. See our corrections guidelines if you need to change the PDF.

Title Adjust the title. Retain tags such as <fixed-case>.

Authors Adjust author names and order to match the PDF.

Abstract Correct abstract if needed. Retain XML formatting tags such as <tex-math>. You may use <b>...</b> for bold, <i>...</i> for italic, and <url>...</url> for URLs.

Verification against PDF Ensure that the new title/authors match the snapshot below. (If there is no snapshot or it is too small, consult the PDF.)

Authors concatenated from the text boxes above:

ALL author names match the snapshot above—including middle initials, hyphens, and accents.

Abstract

We introduce MultiBLiMP 1.0, a massively multilingual benchmark of linguistic minimal pairs, covering 101 languages and 2 types of subject-verb agreement, containing more than 128,000 minimal pairs. Our minimal pairs are created using a fully automated pipeline, leveraging the large-scale linguistic resources of Universal Dependencies and UniMorph. MultiBLiMP 1.0 evaluates abilities of LLMs at an unprecedented multilingual scale, and highlights the shortcomings of the current state-of-the-art in modelling low-resource languages.1

Anthology ID:: 2026.tacl-1.10
Volume:: Transactions of the Association for Computational Linguistics, Volume 14
Month:
Year:: 2026
Address:: Cambridge, MA
Venue:: TACL
SIG:
Publisher:: MIT Press
Note:
Pages:: 193–216
Language:
URL:: https://aclanthology.org/2026.tacl-1.10/
DOI:: 10.1162/tacl.a.600
Bibkey:
Cite (ACL):: Jaap Jumelet, Leonie Weissweiler, Joakim Nivre, and Arianna Bisazza. 2026. MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs. Transactions of the Association for Computational Linguistics, 14:193–216.
Cite (Informal):: MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs (Jumelet et al., TACL 2026)
Copy Citation:
PDF:: https://aclanthology.org/2026.tacl-1.10.pdf

PDF Cite Search Fix data

Export citation

BibTeX
MODS XML
Endnote
Preformatted

@article{jumelet-etal-2026-multiblimp,
    title = "{M}ulti{BL}i{MP} 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs",
    author = "Jumelet, Jaap  and
      Weissweiler, Leonie  and
      Nivre, Joakim  and
      Bisazza, Arianna",
    journal = "Transactions of the Association for Computational Linguistics",
    volume = "14",
    year = "2026",
    address = "Cambridge, MA",
    publisher = "MIT Press",
    url = "https://aclanthology.org/2026.tacl-1.10/",
    doi = "10.1162/tacl.a.600",
    pages = "193--216",
    abstract = "We introduce MultiBLiMP 1.0, a massively multilingual benchmark of linguistic minimal pairs, covering 101 languages and 2 types of subject-verb agreement, containing more than 128,000 minimal pairs. Our minimal pairs are created using a fully automated pipeline, leveraging the large-scale linguistic resources of Universal Dependencies and UniMorph. MultiBLiMP 1.0 evaluates abilities of LLMs at an unprecedented multilingual scale, and highlights the shortcomings of the current state-of-the-art in modelling low-resource languages.1"
}

Download as File

<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="jumelet-etal-2026-multiblimp">
    <titleInfo>
        <title>MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Jaap</namePart>
        <namePart type="family">Jumelet</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Leonie</namePart>
        <namePart type="family">Weissweiler</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Joakim</namePart>
        <namePart type="family">Nivre</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Arianna</namePart>
        <namePart type="family">Bisazza</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>We introduce MultiBLiMP 1.0, a massively multilingual benchmark of linguistic minimal pairs, covering 101 languages and 2 types of subject-verb agreement, containing more than 128,000 minimal pairs. Our minimal pairs are created using a fully automated pipeline, leveraging the large-scale linguistic resources of Universal Dependencies and UniMorph. MultiBLiMP 1.0 evaluates abilities of LLMs at an unprecedented multilingual scale, and highlights the shortcomings of the current state-of-the-art in modelling low-resource languages.1</abstract>
    <identifier type="citekey">jumelet-etal-2026-multiblimp</identifier>
    <identifier type="doi">10.1162/tacl.a.600</identifier>
    <location>
        <url>https://aclanthology.org/2026.tacl-1.10/</url>
    </location>
    <part>
        <date>2026</date>
        <detail type="volume"><number>14</number></detail>
        <extent unit="page">
            <start>193</start>
            <end>216</end>
        </extent>
    </part>
</mods>
</modsCollection>

Download as File

%0 Journal Article
%T MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs
%A Jumelet, Jaap
%A Weissweiler, Leonie
%A Nivre, Joakim
%A Bisazza, Arianna
%J Transactions of the Association for Computational Linguistics
%D 2026
%V 14
%I MIT Press
%C Cambridge, MA
%F jumelet-etal-2026-multiblimp
%X We introduce MultiBLiMP 1.0, a massively multilingual benchmark of linguistic minimal pairs, covering 101 languages and 2 types of subject-verb agreement, containing more than 128,000 minimal pairs. Our minimal pairs are created using a fully automated pipeline, leveraging the large-scale linguistic resources of Universal Dependencies and UniMorph. MultiBLiMP 1.0 evaluates abilities of LLMs at an unprecedented multilingual scale, and highlights the shortcomings of the current state-of-the-art in modelling low-resource languages.1
%R 10.1162/tacl.a.600
%U https://aclanthology.org/2026.tacl-1.10/
%U https://doi.org/10.1162/tacl.a.600
%P 193-216

Download as File

Markdown (Informal)

[MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs](https://aclanthology.org/2026.tacl-1.10/) (Jumelet et al., TACL 2026)

MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs (Jumelet et al., TACL 2026)

ACL

Jaap Jumelet, Leonie Weissweiler, Joakim Nivre, and Arianna Bisazza. 2026. MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs. Transactions of the Association for Computational Linguistics, 14:193–216.