<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="omolaoye-de-melo-2026-gradient">
    <titleInfo>
        <title>Gradient Starvation in Low-Rank Language Model Pretraining</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Victor</namePart>
        <namePart type="given">Adelakun</namePart>
        <namePart type="family">Omolaoye</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Gerard</namePart>
        <namePart type="family">de Melo</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Hung-Yi</namePart>
            <namePart type="family">Lee</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sunipa</namePart>
            <namePart type="family">Dev</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jessy</namePart>
            <namePart type="family">Li</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Desmond</namePart>
            <namePart type="family">Elliott</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-459-0</identifier>
    </relatedItem>
    <abstract>Low-rank (LR) pretraining underperforms full-rank (FR) pretraining, but why the gap persists remains unclear. Using GPT-2 on BabyLM, we show the representational split between them is not fixed at initialisation, spectrum-matched initialisation eliminates the representational gap at step zero, but it’s reasserted within a few training steps, implicating training dynamics over initialisation. We further show gradient dimensionality is a causally separable driver of this split, not merely a restatement of the weight factorization. Restricting or expanding gradient dimensionality of LR models directly shifts representational alignment with FR counterpart independent of weight rank. Finally, optimisation success and representational alignment dissociate, that is, spectral normalisation and other interventions improve perplexity without recovering FR-LR similarity. Also, we observered that representational alignment predicts downstream competence better than perplexity does. These results support gradient starvation as a genuine, partial mechanism, and argue efficient pretraining should be evaluated by both representational trajectory and performance, not solely the latter.</abstract>
    <identifier type="citekey">omolaoye-de-melo-2026-gradient</identifier>
    <location>
        <url>https://aclanthology.org/2026.emnlp-main.360/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>8249</start>
            <end>8266</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Gradient Starvation in Low-Rank Language Model Pretraining
%A Omolaoye, Victor Adelakun
%A de Melo, Gerard
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-459-0
%F omolaoye-de-melo-2026-gradient
%X Low-rank (LR) pretraining underperforms full-rank (FR) pretraining, but why the gap persists remains unclear. Using GPT-2 on BabyLM, we show the representational split between them is not fixed at initialisation, spectrum-matched initialisation eliminates the representational gap at step zero, but it’s reasserted within a few training steps, implicating training dynamics over initialisation. We further show gradient dimensionality is a causally separable driver of this split, not merely a restatement of the weight factorization. Restricting or expanding gradient dimensionality of LR models directly shifts representational alignment with FR counterpart independent of weight rank. Finally, optimisation success and representational alignment dissociate, that is, spectral normalisation and other interventions improve perplexity without recovering FR-LR similarity. Also, we observered that representational alignment predicts downstream competence better than perplexity does. These results support gradient starvation as a genuine, partial mechanism, and argue efficient pretraining should be evaluated by both representational trajectory and performance, not solely the latter.
%U https://aclanthology.org/2026.emnlp-main.360/
%P 8249-8266

Markdown (Informal)

[Gradient Starvation in Low-Rank Language Model Pretraining](https://aclanthology.org/2026.emnlp-main.360/) (Omolaoye & de Melo, EMNLP 2026)

• Gradient Starvation in Low-Rank Language Model Pretraining (Omolaoye & de Melo, EMNLP 2026)

ACL

• Victor Adelakun Omolaoye and Gerard de Melo. 2026. Gradient Starvation in Low-Rank Language Model Pretraining. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 8249–8266, Budapest, Hungary. Association for Computational Linguistics.