<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="torres-futrell-2023-simpler">
    <titleInfo>
        <title>Simpler neural networks prefer subregular languages</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Charles</namePart>
        <namePart type="family">Torres</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Richard</namePart>
        <namePart type="family">Futrell</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2023</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>We apply a continuous relaxation of L₀ regularization (Louizos et al., 2017), which induces sparsity, to study the inductive biases of LSTMs. In particular, we are interested in the patterns of formal languages which are readily learned and expressed by LSTMs. Across a wide range of tests we find sparse LSTMs prefer subregular languages over regular languages and the strength of this preference increases as we increase the pressure for sparsity. Furthermore LSTMs which are trained on subregular languages have fewer non-zero parameters. We conjecture that this subregular bias in LSTMs is related to the cognitive bias for subregular language observed in human phonology which are both downstream of a simplicity bias in a suitable description language.</abstract>
    <identifier type="citekey">torres-futrell-2023-simpler</identifier>
    <identifier type="doi">10.18653/v1/2023.findings-emnlp.112</identifier>
    <location>
        <url>https://aclanthology.org/2023.findings-emnlp.112</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>1651</start>
            <end>1661</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Simpler neural networks prefer subregular languages
%A Torres, Charles
%A Futrell, Richard
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Findings of the Association for Computational Linguistics: EMNLP 2023
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F torres-futrell-2023-simpler
%X We apply a continuous relaxation of L₀ regularization (Louizos et al., 2017), which induces sparsity, to study the inductive biases of LSTMs. In particular, we are interested in the patterns of formal languages which are readily learned and expressed by LSTMs. Across a wide range of tests we find sparse LSTMs prefer subregular languages over regular languages and the strength of this preference increases as we increase the pressure for sparsity. Furthermore LSTMs which are trained on subregular languages have fewer non-zero parameters. We conjecture that this subregular bias in LSTMs is related to the cognitive bias for subregular language observed in human phonology which are both downstream of a simplicity bias in a suitable description language.
%R 10.18653/v1/2023.findings-emnlp.112
%U https://aclanthology.org/2023.findings-emnlp.112
%U https://doi.org/10.18653/v1/2023.findings-emnlp.112
%P 1651-1661

Markdown (Informal)

[Simpler neural networks prefer subregular languages](https://aclanthology.org/2023.findings-emnlp.112) (Torres & Futrell, Findings 2023)

• Simpler neural networks prefer subregular languages (Torres & Futrell, Findings 2023)

ACL

• Charles Torres and Richard Futrell. 2023. Simpler neural networks prefer subregular languages. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1651–1661, Singapore. Association for Computational Linguistics.