<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="zhou-etal-2026-cpsd">
    <titleInfo>
        <title>CPSD: Counterfactual Patches Self-Distillation in RLVR</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Yixiao</namePart>
        <namePart type="family">Zhou</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Dongzhou</namePart>
        <namePart type="family">Cheng</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Hehe</namePart>
        <namePart type="family">Fan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yu</namePart>
        <namePart type="family">Cheng</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2026</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Hung-Yi</namePart>
            <namePart type="family">Lee</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sunipa</namePart>
            <namePart type="family">Dev</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jessy</namePart>
            <namePart type="family">Li</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Desmond</namePart>
            <namePart type="family">Elliott</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-462-0</identifier>
    </relatedItem>
    <abstract>Self-distillation in reinforcement learning with verifiable rewards (RLVR) uses successful in-group rollouts to teach the failed ones, with no external teacher needed. Whether it actually helps turns on three closely related design questions that the canonical method does not handle carefully: how to construct an appropriate teacher context for a given loser, where in the response the distillation loss should act, and what kind of signal it should carry. The canonical Self-Distillation Policy Optimization (SDPO) instantiation answers all three by default: a random in-group winner, a global distillation loss on the whole response, and a Jensen–Shannon divergence on the full top-K distribution. These three defaults compound into a catastrophic late-training failure that short-horizon evaluation hides. We propose CPSD (Counterfactual Patches Self-Distillation), which replaces each default in turn: pick the teacher by minimum embedding-weighted edit distance, localise distillation to an adaptive-stop counterfactual patch beginning at the first divergent token, and replace JS with a Plackett–Luce listwise loss that preserves only the teacher’s ranking. Across six reasoning benchmarks at a fixed 150-step budget, CPSD eliminates the SDPO cliff and lifts average accuracy by +5.7 pp over GRPO on Qwen3-8B and +4.2 pp on Qwen3-4B.</abstract>
    <identifier type="citekey">zhou-etal-2026-cpsd</identifier>
    <location>
        <url>https://aclanthology.org/2026.findings-emnlp.2079/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>44272</start>
            <end>44288</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T CPSD: Counterfactual Patches Self-Distillation in RLVR
%A Zhou, Yixiao
%A Cheng, Dongzhou
%A Fan, Hehe
%A Cheng, Yu
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Findings of the Association for Computational Linguistics: EMNLP 2026
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-462-0
%F zhou-etal-2026-cpsd
%X Self-distillation in reinforcement learning with verifiable rewards (RLVR) uses successful in-group rollouts to teach the failed ones, with no external teacher needed. Whether it actually helps turns on three closely related design questions that the canonical method does not handle carefully: how to construct an appropriate teacher context for a given loser, where in the response the distillation loss should act, and what kind of signal it should carry. The canonical Self-Distillation Policy Optimization (SDPO) instantiation answers all three by default: a random in-group winner, a global distillation loss on the whole response, and a Jensen–Shannon divergence on the full top-K distribution. These three defaults compound into a catastrophic late-training failure that short-horizon evaluation hides. We propose CPSD (Counterfactual Patches Self-Distillation), which replaces each default in turn: pick the teacher by minimum embedding-weighted edit distance, localise distillation to an adaptive-stop counterfactual patch beginning at the first divergent token, and replace JS with a Plackett–Luce listwise loss that preserves only the teacher’s ranking. Across six reasoning benchmarks at a fixed 150-step budget, CPSD eliminates the SDPO cliff and lifts average accuracy by +5.7 pp over GRPO on Qwen3-8B and +4.2 pp on Qwen3-4B.
%U https://aclanthology.org/2026.findings-emnlp.2079/
%P 44272-44288

Markdown (Informal)

[CPSD: Counterfactual Patches Self-Distillation in RLVR](https://aclanthology.org/2026.findings-emnlp.2079/) (Zhou et al., Findings 2026)

• CPSD: Counterfactual Patches Self-Distillation in RLVR (Zhou et al., Findings 2026)

ACL

• Yixiao Zhou, Dongzhou Cheng, Hehe Fan, and Yu Cheng. 2026. CPSD: Counterfactual Patches Self-Distillation in RLVR. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 44272–44288, Budapest, Hungary. Association for Computational Linguistics.