<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="hu-etal-2026-string">
    <titleInfo>
        <title>What Can String Probability Tell Us About Grammaticality?</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Jennifer</namePart>
        <namePart type="family">Hu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ethan</namePart>
        <namePart type="given">Gotlieb</namePart>
        <namePart type="family">Wilcox</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Siyuan</namePart>
        <namePart type="family">Song</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kyle</namePart>
        <namePart type="family">Mahowald</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Roger</namePart>
        <namePart type="given">P</namePart>
        <namePart type="family">Levy</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>What have language models (LMs) learned about grammar? This question remains hotly debated, with major ramifications for linguistic theory. However, since probability and grammaticality are distinct notions in linguistics, it is not obvious what string probabilities can reveal about an LM’s underlying grammatical knowledge. We present a theoretical analysis of the relationship between grammar, meaning, and string probability, based on simple assumptions about the generative process of corpus data. Our framework makes three predictions, which we validate empirically using 280K sentence pairs in English and Chinese: (1) correlation between the probability of strings within minimal pairs, i.e., string pairs with minimal semantic differences; (2) correlation between models’ and humans’ deltas within minimal pairs; and (3) poor separation in probability space between unpaired grammatical and ungrammatical strings. Our analyses give theoretical grounding for using probability to learn about LMs’ structural knowledge, and suggest directions for future work in LM grammatical evaluation.</abstract>
    <identifier type="citekey">hu-etal-2026-string</identifier>
    <identifier type="doi">10.1162/tacl.a.611</identifier>
    <location>
        <url>https://aclanthology.org/2026.tacl-1.7/</url>
    </location>
    <part>
        <date>2026</date>
        <detail type="volume"><number>14</number></detail>
        <extent unit="page">
            <start>124</start>
            <end>146</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T What Can String Probability Tell Us About Grammaticality?
%A Hu, Jennifer
%A Wilcox, Ethan Gotlieb
%A Song, Siyuan
%A Mahowald, Kyle
%A Levy, Roger P.
%J Transactions of the Association for Computational Linguistics
%D 2026
%V 14
%I MIT Press
%C Cambridge, MA
%F hu-etal-2026-string
%X What have language models (LMs) learned about grammar? This question remains hotly debated, with major ramifications for linguistic theory. However, since probability and grammaticality are distinct notions in linguistics, it is not obvious what string probabilities can reveal about an LM’s underlying grammatical knowledge. We present a theoretical analysis of the relationship between grammar, meaning, and string probability, based on simple assumptions about the generative process of corpus data. Our framework makes three predictions, which we validate empirically using 280K sentence pairs in English and Chinese: (1) correlation between the probability of strings within minimal pairs, i.e., string pairs with minimal semantic differences; (2) correlation between models’ and humans’ deltas within minimal pairs; and (3) poor separation in probability space between unpaired grammatical and ungrammatical strings. Our analyses give theoretical grounding for using probability to learn about LMs’ structural knowledge, and suggest directions for future work in LM grammatical evaluation.
%R 10.1162/tacl.a.611
%U https://aclanthology.org/2026.tacl-1.7/
%U https://doi.org/10.1162/tacl.a.611
%P 124-146

Markdown (Informal)

[What Can String Probability Tell Us About Grammaticality?](https://aclanthology.org/2026.tacl-1.7/) (Hu et al., TACL 2026)

• What Can String Probability Tell Us About Grammaticality? (Hu et al., TACL 2026)

ACL

• Jennifer Hu, Ethan Gotlieb Wilcox, Siyuan Song, Kyle Mahowald, and Roger P. Levy. 2026. What Can String Probability Tell Us About Grammaticality?. Transactions of the Association for Computational Linguistics, 14:124–146.