<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="chen-tanaka-ishii-2026-understanding">
    <titleInfo>
        <title>Understanding Benchmark Language Under Weakened Formal Semantics</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Haoyang</namePart>
        <namePart type="family">Chen</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kumiko</namePart>
        <namePart type="family">Tanaka-Ishii</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>State-of-the-art NLP benchmarks require interpretation of natural-language that specify conditions, procedures, and exceptions, often relying on implicit assumptions and external knowledge. Constructing complete semantic representations with proof-theoretic guarantees is frequently impractical at scale, and purely text-based reasoning offers limited means of inspection. This paper asks how much understanding of benchmark language can be achieved when formal semantic guarantees are weakened. We investigate this question by extracting computables: executable representations whose runtime behavior provides operational evidence of semantic adequacy, including executability, execution traces, and runtime failures. We induce and iteratively refine computables for benchmark instances using retrieval from external knowledge. Across mathematical reasoning, multi-step reasoning, causal inference, and rule- and exception-heavy legal and biomedical benchmarks, we find that the proposed approach consistently exceeds text-only reasoning and one-shot code execution. Beyond accuracy, our analyses show that these computables provide scalable, inspectable semantic evidence: they expose conditions and exceptions benchmark language forces into executable form, offering a practical bridge between proof-oriented semantics and purely textual reasoning.</abstract>
    <identifier type="citekey">chen-tanaka-ishii-2026-understanding</identifier>
    <identifier type="doi">10.1162/tacl.a.789</identifier>
    <location>
        <url>https://aclanthology.org/2026.tacl-1.91/</url>
    </location>
    <part>
        <date>2026</date>
        <detail type="volume"><number>14</number></detail>
        <extent unit="page">
            <start>2012</start>
            <end>2032</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T Understanding Benchmark Language Under Weakened Formal Semantics
%A Chen, Haoyang
%A Tanaka-Ishii, Kumiko
%J Transactions of the Association for Computational Linguistics
%D 2026
%V 14
%I MIT Press
%C Cambridge, MA
%F chen-tanaka-ishii-2026-understanding
%X State-of-the-art NLP benchmarks require interpretation of natural-language that specify conditions, procedures, and exceptions, often relying on implicit assumptions and external knowledge. Constructing complete semantic representations with proof-theoretic guarantees is frequently impractical at scale, and purely text-based reasoning offers limited means of inspection. This paper asks how much understanding of benchmark language can be achieved when formal semantic guarantees are weakened. We investigate this question by extracting computables: executable representations whose runtime behavior provides operational evidence of semantic adequacy, including executability, execution traces, and runtime failures. We induce and iteratively refine computables for benchmark instances using retrieval from external knowledge. Across mathematical reasoning, multi-step reasoning, causal inference, and rule- and exception-heavy legal and biomedical benchmarks, we find that the proposed approach consistently exceeds text-only reasoning and one-shot code execution. Beyond accuracy, our analyses show that these computables provide scalable, inspectable semantic evidence: they expose conditions and exceptions benchmark language forces into executable form, offering a practical bridge between proof-oriented semantics and purely textual reasoning.
%R 10.1162/tacl.a.789
%U https://aclanthology.org/2026.tacl-1.91/
%U https://doi.org/10.1162/tacl.a.789
%P 2012-2032

Markdown (Informal)

[Understanding Benchmark Language Under Weakened Formal Semantics](https://aclanthology.org/2026.tacl-1.91/) (Chen & Tanaka-Ishii, TACL 2026)

• Understanding Benchmark Language Under Weakened Formal Semantics (Chen & Tanaka-Ishii, TACL 2026)

ACL

• Haoyang Chen and Kumiko Tanaka-Ishii. 2026. Understanding Benchmark Language Under Weakened Formal Semantics. Transactions of the Association for Computational Linguistics, 14:2012–2032.