On-Policy Delta Distillation for Multilingual Math Reasoning

Byeongho Heo; Jaehui Hwang; Sangdoo Yun; Dongyoon Han

On-Policy Delta Distillation for Multilingual Math Reasoning

Byeongho Heo, Jaehui Hwang, Sangdoo Yun, Dongyoon Han

Correct Metadata for

Use this form to create a GitHub issue with structured data describing the correction. You will need a GitHub account. Once you create that issue, the correction will be reviewed by a staff member.

⚠️ Mobile Users: Submitting this form to create a new issue will only work with github.com, not the GitHub Mobile app.

Important: The Anthology treat PDFs as authoritative. Please use this form only to correct data that is out of line with the PDF. See our corrections guidelines if you need to change the PDF.

Title Adjust the title. Retain tags such as <fixed-case>.

Authors Adjust author names and order to match the PDF.

Abstract Correct abstract if needed. Retain XML formatting tags such as <tex-math>. You may use ... for bold, ... for italic, ... for underline, <sc>...</sc> for small-caps, <tt>...<tt> for typewriter text, <url>...</url> for URLs, <a href=...> for hyperlinks, and <par/> for paragraph breaks.

Verification against PDF Ensure that the new title/authors match the snapshot below. (If there is no snapshot or it is too small, consult the PDF.)

Authors concatenated from the text boxes above:

ALL author names match the snapshot above—including middle initials, hyphens, and accents.

Abstract

On-Policy Distillation (OPD) is emerging as a promising alternative to reinforcement learning for LLM post-training, yet its effectiveness in multilingual settings remains underexplored. We study OPD and its advanced variant, OnPolicy Delta Distillation (OPD2 ), for mathematical reasoning in English, Korean, and Japanese. OPD2 improves OPD by using the probability gap between a post-trained teacher and its base model as the learning signal. Experiments with Qwen3 show that OPD2 consistently outperforms the original OPD, with particularly strong improvements in Korean and Japanese. We further compare multilingual OPD with English-only OPD to analyze the effect of multilingual training on OPD.

Anthology ID:: 2026.oracle-main.7
Volume:: The Proceedings of the Workshop on Open Reasoning Across Cultures Languages (ORACLE)
Month:: October
Year:: 2026
Address:: Budapest, Hungary
Editors:: Leonardo Ranaldi, Giulia Pucci, Federico Ranaldi, Marco Valentino, Alexandra Birch, Fabio Massimo Zanzotto
Venues:: ORACLE | WS
SIG:
Publisher:: Association for Computational Linguistics
Note:
Pages:: 99–106
Language:
URL:: https://aclanthology.org/2026.oracle-main.7/
DOI:
Bibkey:
Cite (ACL):: Byeongho Heo, Jaehui Hwang, Sangdoo Yun, and Dongyoon Han. 2026. On-Policy Delta Distillation for Multilingual Math Reasoning. In The Proceedings of the Workshop on Open Reasoning Across Cultures Languages (ORACLE), pages 99–106, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):: On-Policy Delta Distillation for Multilingual Math Reasoning (Heo et al., ORACLE 2026)
Copy Citation:
PDF:: https://aclanthology.org/2026.oracle-main.7.pdf

PDF Cite Search Fix data

Export citation

BibTeX
MODS XML
Endnote
Preformatted

@inproceedings{heo-etal-2026-policy,
    title = "On-Policy Delta Distillation for Multilingual Math Reasoning",
    author = "Heo, Byeongho  and
      Hwang, Jaehui  and
      Yun, Sangdoo  and
      Han, Dongyoon",
    editor = "Ranaldi, Leonardo  and
      Pucci, Giulia  and
      Ranaldi, Federico  and
      Valentino, Marco  and
      Birch, Alexandra  and
      Zanzotto, Fabio Massimo",
    booktitle = "The Proceedings of the Workshop on Open Reasoning Across Cultures Languages ({ORACLE})",
    month = oct,
    year = "2026",
    address = "Budapest, Hungary",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.oracle-main.7/",
    pages = "99--106",
    ISBN = "979-8-89176-483-5",
    abstract = "On-Policy Distillation (OPD) is emerging as a promising alternative to reinforcement learning for LLM post-training, yet its effectiveness in multilingual settings remains underexplored. We study OPD and its advanced variant, OnPolicy Delta Distillation (OPD2 ), for mathematical reasoning in English, Korean, and Japanese. OPD2 improves OPD by using the probability gap between a post-trained teacher and its base model as the learning signal. Experiments with Qwen3 show that OPD2 consistently outperforms the original OPD, with particularly strong improvements in Korean and Japanese. We further compare multilingual OPD with English-only OPD to analyze the effect of multilingual training on OPD."
}

Download as File

<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="heo-etal-2026-policy">
    <titleInfo>
        <title>On-Policy Delta Distillation for Multilingual Math Reasoning</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Byeongho</namePart>
        <namePart type="family">Heo</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jaehui</namePart>
        <namePart type="family">Hwang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Sangdoo</namePart>
        <namePart type="family">Yun</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Dongyoon</namePart>
        <namePart type="family">Han</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>The Proceedings of the Workshop on Open Reasoning Across Cultures Languages (ORACLE)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Leonardo</namePart>
            <namePart type="family">Ranaldi</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Giulia</namePart>
            <namePart type="family">Pucci</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Federico</namePart>
            <namePart type="family">Ranaldi</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Marco</namePart>
            <namePart type="family">Valentino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Alexandra</namePart>
            <namePart type="family">Birch</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Fabio</namePart>
            <namePart type="given">Massimo</namePart>
            <namePart type="family">Zanzotto</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-483-5</identifier>
    </relatedItem>
    <abstract>On-Policy Distillation (OPD) is emerging as a promising alternative to reinforcement learning for LLM post-training, yet its effectiveness in multilingual settings remains underexplored. We study OPD and its advanced variant, OnPolicy Delta Distillation (OPD2 ), for mathematical reasoning in English, Korean, and Japanese. OPD2 improves OPD by using the probability gap between a post-trained teacher and its base model as the learning signal. Experiments with Qwen3 show that OPD2 consistently outperforms the original OPD, with particularly strong improvements in Korean and Japanese. We further compare multilingual OPD with English-only OPD to analyze the effect of multilingual training on OPD.</abstract>
    <identifier type="citekey">heo-etal-2026-policy</identifier>
    <location>
        <url>https://aclanthology.org/2026.oracle-main.7/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>99</start>
            <end>106</end>
        </extent>
    </part>
</mods>
</modsCollection>

Download as File

%0 Conference Proceedings
%T On-Policy Delta Distillation for Multilingual Math Reasoning
%A Heo, Byeongho
%A Hwang, Jaehui
%A Yun, Sangdoo
%A Han, Dongyoon
%Y Ranaldi, Leonardo
%Y Pucci, Giulia
%Y Ranaldi, Federico
%Y Valentino, Marco
%Y Birch, Alexandra
%Y Zanzotto, Fabio Massimo
%S The Proceedings of the Workshop on Open Reasoning Across Cultures Languages (ORACLE)
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-483-5
%F heo-etal-2026-policy
%X On-Policy Distillation (OPD) is emerging as a promising alternative to reinforcement learning for LLM post-training, yet its effectiveness in multilingual settings remains underexplored. We study OPD and its advanced variant, OnPolicy Delta Distillation (OPD2 ), for mathematical reasoning in English, Korean, and Japanese. OPD2 improves OPD by using the probability gap between a post-trained teacher and its base model as the learning signal. Experiments with Qwen3 show that OPD2 consistently outperforms the original OPD, with particularly strong improvements in Korean and Japanese. We further compare multilingual OPD with English-only OPD to analyze the effect of multilingual training on OPD.
%U https://aclanthology.org/2026.oracle-main.7/
%P 99-106

Download as File

Markdown (Informal)

[On-Policy Delta Distillation for Multilingual Math Reasoning](https://aclanthology.org/2026.oracle-main.7/) (Heo et al., ORACLE 2026)

On-Policy Delta Distillation for Multilingual Math Reasoning (Heo et al., ORACLE 2026)

ACL

Byeongho Heo, Jaehui Hwang, Sangdoo Yun, and Dongyoon Han. 2026. On-Policy Delta Distillation for Multilingual Math Reasoning. In The Proceedings of the Workshop on Open Reasoning Across Cultures Languages (ORACLE), pages 99–106, Budapest, Hungary. Association for Computational Linguistics.