<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="janarthan-etal-2026-transformers">
    <titleInfo>
        <title>When transformers learn “impossible” languages, what do they learn?</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Ram</namePart>
        <namePart type="family">Janarthan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Coleman</namePart>
        <namePart type="family">Haley</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Sharon</namePart>
        <namePart type="family">Goldwater</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 30th Conference on Computational Natural Language Learning</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Claire</namePart>
            <namePart type="family">Bonial</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yevgeni</namePart>
            <namePart type="family">Berzak</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">San Diego, California, USA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-410-1</identifier>
    </relatedItem>
    <abstract>Recent work suggests that transformer language models show a bias towards human languages over unnatural (”impossible”) languages argued to be unacquirable by humans. However, this literature has largely based these claims on differences in sample efficiency and test-set perplexity, rather than on direct evaluations of the linguistic capacities that could plausibly explain non-attestation in human languages. We evaluate two theoretically motivated linking hypotheses: impossibility arising from deficiencies in grammatical sensitivity or generative production. Using GPT-2 style models trained on perturbed “impossible” variants of English, we measure sensitivity to grammaticality using BLiMP minimal pairs, finding that model performance exhibits only gradual degradation, mediated by the language’s information locality. In contrast, these models exhibited pronounced failures in generation, producing substantially fewer high-quality sentences at longer lengths. Together, these results suggest generative deficiency and transmission failures as a plausible linking hypothesis between language model behaviour and non-attestation of impossible languages.</abstract>
    <identifier type="citekey">janarthan-etal-2026-transformers</identifier>
    <location>
        <url>https://aclanthology.org/2026.conll-main.24/</url>
    </location>
    <part>
        <date>2026-07</date>
        <extent unit="page">
            <start>421</start>
            <end>434</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T When transformers learn “impossible” languages, what do they learn?
%A Janarthan, Ram
%A Haley, Coleman
%A Goldwater, Sharon
%Y Bonial, Claire
%Y Berzak, Yevgeni
%S Proceedings of the 30th Conference on Computational Natural Language Learning
%D 2026
%8 July
%I Association for Computational Linguistics
%C San Diego, California, USA
%@ 979-8-89176-410-1
%F janarthan-etal-2026-transformers
%X Recent work suggests that transformer language models show a bias towards human languages over unnatural (”impossible”) languages argued to be unacquirable by humans. However, this literature has largely based these claims on differences in sample efficiency and test-set perplexity, rather than on direct evaluations of the linguistic capacities that could plausibly explain non-attestation in human languages. We evaluate two theoretically motivated linking hypotheses: impossibility arising from deficiencies in grammatical sensitivity or generative production. Using GPT-2 style models trained on perturbed “impossible” variants of English, we measure sensitivity to grammaticality using BLiMP minimal pairs, finding that model performance exhibits only gradual degradation, mediated by the language’s information locality. In contrast, these models exhibited pronounced failures in generation, producing substantially fewer high-quality sentences at longer lengths. Together, these results suggest generative deficiency and transmission failures as a plausible linking hypothesis between language model behaviour and non-attestation of impossible languages.
%U https://aclanthology.org/2026.conll-main.24/
%P 421-434

Markdown (Informal)

[When transformers learn “impossible” languages, what do they learn?](https://aclanthology.org/2026.conll-main.24/) (Janarthan et al., CoNLL 2026)

• When transformers learn “impossible” languages, what do they learn? (Janarthan et al., CoNLL 2026)

ACL

• Ram Janarthan, Coleman Haley, and Sharon Goldwater. 2026. When transformers learn “impossible” languages, what do they learn?. In Proceedings of the 30th Conference on Computational Natural Language Learning, pages 421–434, San Diego, California, USA. Association for Computational Linguistics.