<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="harish-2026-sample">
    <titleInfo>
        <title>Sample-Matched Comparisons Overstate Test-Time Compute Gains</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Srinivas</namePart>
        <namePart type="family">Harish</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>The Proceedings of the Workshop on Open Reasoning Across Cultures Languages (ORACLE)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Leonardo</namePart>
            <namePart type="family">Ranaldi</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Giulia</namePart>
            <namePart type="family">Pucci</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Federico</namePart>
            <namePart type="family">Ranaldi</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Marco</namePart>
            <namePart type="family">Valentino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Alexandra</namePart>
            <namePart type="family">Birch</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Fabio</namePart>
            <namePart type="given">Massimo</namePart>
            <namePart type="family">Zanzotto</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-483-5</identifier>
    </relatedItem>
    <abstract>Inference-time methods buy accuracy with extra compute: sample several reasoning traces and combine them. They are compared at a matched number of samples, and that protocol hides two things. Samples do not cost equal tokens, so matched samples is not matched budget; and it never asks whether a cheaper source of diversity already reaches the same ceiling. We measure both on cross-lingual aggregation, which votes across traces generated in different languages, using 32,992 traces from Qwen38B across 11 languages on two item-parallel corpora. At a matched trace budget on MGSM, one trace from each of the eight strongest languages buys no ceiling and no vote that resampling English eight times does not: the per-item oracle is 0.992 for samples against 0.989 for languages, the vote 0.966 against 0.956, and the language ballot costs 1.09× the tokens. Over the full pool the eighty traces from the other ten languages solve 0 of 250 items that English’s own eight do not. On MMLU-ProX-Lite, far from saturation, the languages do hold coverage English never reaches (0.895 against 0.685 over the full 88-trace pool against English’s eight, 26 items) — and at a matched budget of eight traces the language ballot there is still the weaker vote (0.453 against 0.487). What blocks the conversion is not error correlation, which is low: an independence model built from the measured marginals over-predicts the vote by 5.4 and 48.0 points, because the independence sits on items every language already answers and vanishes on the items that decide the score, where wrong answers concentrate on a single value (0.464 of erring languages). Three selectors computable from the released traces without gold access all fail to beat the best single language. We release both pools with per-trace token counts so any aggregation or routing rule can be scored against these ceilings by re-scoring rather than generating.</abstract>
    <identifier type="citekey">harish-2026-sample</identifier>
    <location>
        <url>https://aclanthology.org/2026.oracle-main.23/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>339</start>
            <end>353</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Sample-Matched Comparisons Overstate Test-Time Compute Gains
%A Harish, Srinivas
%Y Ranaldi, Leonardo
%Y Pucci, Giulia
%Y Ranaldi, Federico
%Y Valentino, Marco
%Y Birch, Alexandra
%Y Zanzotto, Fabio Massimo
%S The Proceedings of the Workshop on Open Reasoning Across Cultures Languages (ORACLE)
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-483-5
%F harish-2026-sample
%X Inference-time methods buy accuracy with extra compute: sample several reasoning traces and combine them. They are compared at a matched number of samples, and that protocol hides two things. Samples do not cost equal tokens, so matched samples is not matched budget; and it never asks whether a cheaper source of diversity already reaches the same ceiling. We measure both on cross-lingual aggregation, which votes across traces generated in different languages, using 32,992 traces from Qwen38B across 11 languages on two item-parallel corpora. At a matched trace budget on MGSM, one trace from each of the eight strongest languages buys no ceiling and no vote that resampling English eight times does not: the per-item oracle is 0.992 for samples against 0.989 for languages, the vote 0.966 against 0.956, and the language ballot costs 1.09× the tokens. Over the full pool the eighty traces from the other ten languages solve 0 of 250 items that English’s own eight do not. On MMLU-ProX-Lite, far from saturation, the languages do hold coverage English never reaches (0.895 against 0.685 over the full 88-trace pool against English’s eight, 26 items) — and at a matched budget of eight traces the language ballot there is still the weaker vote (0.453 against 0.487). What blocks the conversion is not error correlation, which is low: an independence model built from the measured marginals over-predicts the vote by 5.4 and 48.0 points, because the independence sits on items every language already answers and vanishes on the items that decide the score, where wrong answers concentrate on a single value (0.464 of erring languages). Three selectors computable from the released traces without gold access all fail to beat the best single language. We release both pools with per-trace token counts so any aggregation or routing rule can be scored against these ceilings by re-scoring rather than generating.
%U https://aclanthology.org/2026.oracle-main.23/
%P 339-353

Markdown (Informal)

[Sample-Matched Comparisons Overstate Test-Time Compute Gains](https://aclanthology.org/2026.oracle-main.23/) (Harish, ORACLE 2026)

• Sample-Matched Comparisons Overstate Test-Time Compute Gains (Harish, ORACLE 2026)

ACL

• Srinivas Harish. 2026. Sample-Matched Comparisons Overstate Test-Time Compute Gains. In The Proceedings of the Workshop on Open Reasoning Across Cultures Languages (ORACLE), pages 339–353, Budapest, Hungary. Association for Computational Linguistics.