<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="srivastav-2026-causal">
    <titleInfo>
        <title>Causal, Hybrid, and Ternary Transformers under a Restricted Data Budget</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Arnav</namePart>
        <namePart type="family">Srivastav</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Second BabyLM Workshop</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Leshem</namePart>
            <namePart type="family">Choshen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ryan</namePart>
            <namePart type="family">Cotterell</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Mustafa</namePart>
            <namePart type="given">Omer</namePart>
            <namePart type="family">Gul</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jaap</namePart>
            <namePart type="family">Jumelet</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Tal</namePart>
            <namePart type="family">Linzen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Aaron</namePart>
            <namePart type="family">Mueller</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Suchir</namePart>
            <namePart type="family">Salhan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Raj</namePart>
            <namePart type="given">Sanjay</namePart>
            <namePart type="family">Shah</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Alex</namePart>
            <namePart type="family">Warstadt</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ethan</namePart>
            <namePart type="given">Gotlieb</namePart>
            <namePart type="family">Wilcox</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-475-0</identifier>
    </relatedItem>
    <abstract>We evaluate five decoder-only Transformers trained in the BabyLM 2026 Strict-Small setting. In a closely matched training comparison, Causal-30M and Hybrid-30M share architecture, initialization, data, tokenization, optimization settings, seed, and training exposure. Causal-30M reaches lower held-out causal next-token loss. Their benchmark results come from different evaluation pipelines—server-held-out evaluation for Causal-30M and preliminary local evaluation for Hybrid-30M—so we report them descriptively and do not treat their benchmark differences as an objective-level ranking. A larger 64.57M-parameter hybrid configuration improves the preliminary NLP Average and substantially improves Entity Tracking, but its gains are not uniform across evaluation categories.\par Separately, we compare architecture- and exposure-matched dense and fake-quantized ternary models. In this single-seed comparison, Ternary-30M averages 42.90 across the eight categories measured for both models, against 43.45 for Dense-30M, retaining 98.73% of the dense model’s measured aggregate. The two models use different peak learning rates, so this result describes the tested training recipes, not the effect of ternarization alone. Overall, the experiments show benchmark-specific trade-offs across objectives, model configurations, and weight representations. The ternary experiment measures capability retention under fake quantization and does not assess packed storage or specialized low-bit inference.</abstract>
    <identifier type="citekey">srivastav-2026-causal</identifier>
    <location>
        <url>https://aclanthology.org/2026.babylm-main.23/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>289</start>
            <end>298</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Causal, Hybrid, and Ternary Transformers under a Restricted Data Budget
%A Srivastav, Arnav
%Y Choshen, Leshem
%Y Cotterell, Ryan
%Y Gul, Mustafa Omer
%Y Jumelet, Jaap
%Y Linzen, Tal
%Y Mueller, Aaron
%Y Salhan, Suchir
%Y Shah, Raj Sanjay
%Y Warstadt, Alex
%Y Wilcox, Ethan Gotlieb
%S Proceedings of the Second BabyLM Workshop
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-475-0
%F srivastav-2026-causal
%X We evaluate five decoder-only Transformers trained in the BabyLM 2026 Strict-Small setting. In a closely matched training comparison, Causal-30M and Hybrid-30M share architecture, initialization, data, tokenization, optimization settings, seed, and training exposure. Causal-30M reaches lower held-out causal next-token loss. Their benchmark results come from different evaluation pipelines—server-held-out evaluation for Causal-30M and preliminary local evaluation for Hybrid-30M—so we report them descriptively and do not treat their benchmark differences as an objective-level ranking. A larger 64.57M-parameter hybrid configuration improves the preliminary NLP Average and substantially improves Entity Tracking, but its gains are not uniform across evaluation categories.\par Separately, we compare architecture- and exposure-matched dense and fake-quantized ternary models. In this single-seed comparison, Ternary-30M averages 42.90 across the eight categories measured for both models, against 43.45 for Dense-30M, retaining 98.73% of the dense model’s measured aggregate. The two models use different peak learning rates, so this result describes the tested training recipes, not the effect of ternarization alone. Overall, the experiments show benchmark-specific trade-offs across objectives, model configurations, and weight representations. The ternary experiment measures capability retention under fake quantization and does not assess packed storage or specialized low-bit inference.
%U https://aclanthology.org/2026.babylm-main.23/
%P 289-298

Markdown (Informal)

[Causal, Hybrid, and Ternary Transformers under a Restricted Data Budget](https://aclanthology.org/2026.babylm-main.23/) (Srivastav, BabyLM 2026)

• Causal, Hybrid, and Ternary Transformers under a Restricted Data Budget (Srivastav, BabyLM 2026)

ACL

• Arnav Srivastav. 2026. Causal, Hybrid, and Ternary Transformers under a Restricted Data Budget. In Proceedings of the Second BabyLM Workshop, pages 289–298, Budapest, Hungary. Association for Computational Linguistics.