<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="lu-etal-2026-beyond-one">
    <titleInfo>
        <title>Beyond One-Hot Replay: Self-Distillation Reduces Target Variance in Continual Pre-Training</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Peng</namePart>
        <namePart type="family">Lu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Sijie</namePart>
        <namePart type="family">Dong</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Xiaoyun</namePart>
        <namePart type="family">Feng</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Xuanwei</namePart>
        <namePart type="family">Hu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Xin</namePart>
        <namePart type="family">Zhang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2026</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Hung-Yi</namePart>
            <namePart type="family">Lee</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sunipa</namePart>
            <namePart type="family">Dev</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jessy</namePart>
            <namePart type="family">Li</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Desmond</namePart>
            <namePart type="family">Elliott</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-462-0</identifier>
    </relatedItem>
    <abstract>Continual pre-training (CPT) adapts a pretrained language model to a target domain, but it often degrades general capabilities even when general-data replay is used to mitigate forgetting. We revisit this failure through replay supervision: prior CPT replay mainly tunes data mixtures and token budgets while applying one-hot next-token prediction (NTP) uniformly to domain and replay tokens. This target is natural for learning domain text, but on replay tokens it introduces target variance: it is a noisy, corpus-dependent proxy for preserving the frozen base model’s conditional behavior. We study self-distilled replay: domain tokens keep NTP, while replay tokens are trained to match the frozen base model distribution. Diagnostics show that this target substitution lowers target error, produces more directionally consistent replay gradients, and reduces base-behavior drift. Empirically, self-distilled replay improves the SmolLM3 domain–general trade-off over matched NTP replay across diverse token budgets and replay ratios. It also remains effective for Qwen3 models whose pretraining mixtures are unavailable: under the same mismatched, non-source replay setting, it improves both general score and PubMedQA in all eight matched comparisons. Overall, these results suggest supervising CPT replay to preserve base behavior, rather than fit empirical next tokens, as a way to reduce target variance.</abstract>
    <identifier type="citekey">lu-etal-2026-beyond-one</identifier>
    <location>
        <url>https://aclanthology.org/2026.findings-emnlp.1299/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>27473</start>
            <end>27491</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Beyond One-Hot Replay: Self-Distillation Reduces Target Variance in Continual Pre-Training
%A Lu, Peng
%A Dong, Sijie
%A Feng, Xiaoyun
%A Hu, Xuanwei
%A Zhang, Xin
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Findings of the Association for Computational Linguistics: EMNLP 2026
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-462-0
%F lu-etal-2026-beyond-one
%X Continual pre-training (CPT) adapts a pretrained language model to a target domain, but it often degrades general capabilities even when general-data replay is used to mitigate forgetting. We revisit this failure through replay supervision: prior CPT replay mainly tunes data mixtures and token budgets while applying one-hot next-token prediction (NTP) uniformly to domain and replay tokens. This target is natural for learning domain text, but on replay tokens it introduces target variance: it is a noisy, corpus-dependent proxy for preserving the frozen base model’s conditional behavior. We study self-distilled replay: domain tokens keep NTP, while replay tokens are trained to match the frozen base model distribution. Diagnostics show that this target substitution lowers target error, produces more directionally consistent replay gradients, and reduces base-behavior drift. Empirically, self-distilled replay improves the SmolLM3 domain–general trade-off over matched NTP replay across diverse token budgets and replay ratios. It also remains effective for Qwen3 models whose pretraining mixtures are unavailable: under the same mismatched, non-source replay setting, it improves both general score and PubMedQA in all eight matched comparisons. Overall, these results suggest supervising CPT replay to preserve base behavior, rather than fit empirical next tokens, as a way to reduce target variance.
%U https://aclanthology.org/2026.findings-emnlp.1299/
%P 27473-27491

Markdown (Informal)

[Beyond One-Hot Replay: Self-Distillation Reduces Target Variance in Continual Pre-Training](https://aclanthology.org/2026.findings-emnlp.1299/) (Lu et al., Findings 2026)

• Beyond One-Hot Replay: Self-Distillation Reduces Target Variance in Continual Pre-Training (Lu et al., Findings 2026)

ACL

• Peng Lu, Sijie Dong, Xiaoyun Feng, Xuanwei Hu, and Xin Zhang. 2026. Beyond One-Hot Replay: Self-Distillation Reduces Target Variance in Continual Pre-Training. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 27473–27491, Budapest, Hungary. Association for Computational Linguistics.