<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="rosenthal-etal-2026-unexplored">
    <titleInfo>
        <title>Unexplored flaws in multiple-choice VQA make benchmarking unreliable</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Fabio</namePart>
        <namePart type="family">Rosenthal</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Sebastian</namePart>
        <namePart type="family">Schmidt</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Dr.</namePart>
        <namePart type="given">Thorsten</namePart>
        <namePart type="family">Graf</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Thorsten</namePart>
        <namePart type="family">Bagdonat</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Stephan</namePart>
        <namePart type="family">Günnemann</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Leo</namePart>
        <namePart type="family">Schwinn</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2026</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Hung-Yi</namePart>
            <namePart type="family">Lee</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sunipa</namePart>
            <namePart type="family">Dev</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jessy</namePart>
            <namePart type="family">Li</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Desmond</namePart>
            <namePart type="family">Elliott</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-462-0</identifier>
    </relatedItem>
    <abstract>Previous works identify sensitivity to option order as a key issue in MC-VQA evaluation and propose protocols to mitigate this effect. We show that such mitigation is insufficient to ensure the validity of MC-VQA as a reliable benchmark for MLLMs: performance remains highly sensitive to semantically neutral prompt format choices that are not controlled by current benchmarks. In a large-scale study spanning seven MLLMs and five MC-VQA datasets, we find frequent rank reversals even under order-invariant evaluation. These reversals arise when we systematically vary option ID sets, delimiters, and separators, yielding 48 semantically equivalent prompt formats. Mechanistic analyses trace this instability to low-level language modeling effects: tokenizer-induced fusion or removal of option ID tokens introduces corrupted option ID tokens into the input sequence, while the choice of option ID sets directly affects the reliability of attention patterns for option selection. Accordingly, MC-VQA rankings correlate weakly with open-ended evaluation, indicating that MC-VQA reflects option selection dynamics in addition to multimodal reasoning. These findings identify prompt formatting as a major, previously under-controlled confounder in MC-VQA benchmarking and motivate evaluation protocols that explicitly control prompt format sensitivity.</abstract>
    <identifier type="citekey">rosenthal-etal-2026-unexplored</identifier>
    <location>
        <url>https://aclanthology.org/2026.findings-emnlp.162/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>3412</start>
            <end>3451</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Unexplored flaws in multiple-choice VQA make benchmarking unreliable
%A Rosenthal, Fabio
%A Schmidt, Sebastian
%A Graf, Dr. Thorsten
%A Bagdonat, Thorsten
%A Günnemann, Stephan
%A Schwinn, Leo
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Findings of the Association for Computational Linguistics: EMNLP 2026
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-462-0
%F rosenthal-etal-2026-unexplored
%X Previous works identify sensitivity to option order as a key issue in MC-VQA evaluation and propose protocols to mitigate this effect. We show that such mitigation is insufficient to ensure the validity of MC-VQA as a reliable benchmark for MLLMs: performance remains highly sensitive to semantically neutral prompt format choices that are not controlled by current benchmarks. In a large-scale study spanning seven MLLMs and five MC-VQA datasets, we find frequent rank reversals even under order-invariant evaluation. These reversals arise when we systematically vary option ID sets, delimiters, and separators, yielding 48 semantically equivalent prompt formats. Mechanistic analyses trace this instability to low-level language modeling effects: tokenizer-induced fusion or removal of option ID tokens introduces corrupted option ID tokens into the input sequence, while the choice of option ID sets directly affects the reliability of attention patterns for option selection. Accordingly, MC-VQA rankings correlate weakly with open-ended evaluation, indicating that MC-VQA reflects option selection dynamics in addition to multimodal reasoning. These findings identify prompt formatting as a major, previously under-controlled confounder in MC-VQA benchmarking and motivate evaluation protocols that explicitly control prompt format sensitivity.
%U https://aclanthology.org/2026.findings-emnlp.162/
%P 3412-3451

Markdown (Informal)

[Unexplored flaws in multiple-choice VQA make benchmarking unreliable](https://aclanthology.org/2026.findings-emnlp.162/) (Rosenthal et al., Findings 2026)

• Unexplored flaws in multiple-choice VQA make benchmarking unreliable (Rosenthal et al., Findings 2026)

ACL

• Fabio Rosenthal, Sebastian Schmidt, Dr. Thorsten Graf, Thorsten Bagdonat, Stephan Günnemann, and Leo Schwinn. 2026. Unexplored flaws in multiple-choice VQA make benchmarking unreliable. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 3412–3451, Budapest, Hungary. Association for Computational Linguistics.