<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="glazyrina-etal-2026-soft">
    <titleInfo>
        <title>Soft Sequence Policy Optimization</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Svetlana</namePart>
        <namePart type="family">Glazyrina</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Maksim</namePart>
        <namePart type="family">Kryzhanovskiy</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Roman</namePart>
        <namePart type="family">Ischenko</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2026</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Hung-Yi</namePart>
            <namePart type="family">Lee</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sunipa</namePart>
            <namePart type="family">Dev</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jessy</namePart>
            <namePart type="family">Li</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Desmond</namePart>
            <namePart type="family">Elliott</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-462-0</identifier>
    </relatedItem>
    <abstract>A significant portion of recent research on Large Language Model (LLM) alignment focuses on developing new policy optimization methods based on Group Relative Policy Optimization (GRPO). Two prominent directions have emerged: (i) a shift toward sequence-level importance sampling weights that better align with the sequence-level rewards used in many tasks, and (ii) alternatives to the PPO-style clipping that aim to avoid the associated loss of training signal and entropy collapse. We introduce Soft Sequence Policy Optimization, an off-policy reinforcement learning objective that incorporates soft gating functions over token-level probability ratios within sequence-level importance weights. We provide theoretical motivation for SSPO and investigate practical modifications to improve optimization behavior. Empirically, we demonstrate that SSPO improves training stability and performance both in mathematical reasoning and coding tasks.</abstract>
    <identifier type="citekey">glazyrina-etal-2026-soft</identifier>
    <identifier type="doi">10.18653/2026.findings-emnlp.2472</identifier>
    <location>
        <url>https://aclanthology.org/2026.findings-emnlp.2472/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>53191</start>
            <end>53201</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Soft Sequence Policy Optimization
%A Glazyrina, Svetlana
%A Kryzhanovskiy, Maksim
%A Ischenko, Roman
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Findings of the Association for Computational Linguistics: EMNLP 2026
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-462-0
%F glazyrina-etal-2026-soft
%X A significant portion of recent research on Large Language Model (LLM) alignment focuses on developing new policy optimization methods based on Group Relative Policy Optimization (GRPO). Two prominent directions have emerged: (i) a shift toward sequence-level importance sampling weights that better align with the sequence-level rewards used in many tasks, and (ii) alternatives to the PPO-style clipping that aim to avoid the associated loss of training signal and entropy collapse. We introduce Soft Sequence Policy Optimization, an off-policy reinforcement learning objective that incorporates soft gating functions over token-level probability ratios within sequence-level importance weights. We provide theoretical motivation for SSPO and investigate practical modifications to improve optimization behavior. Empirically, we demonstrate that SSPO improves training stability and performance both in mathematical reasoning and coding tasks.
%R 10.18653/2026.findings-emnlp.2472
%U https://aclanthology.org/2026.findings-emnlp.2472/
%U https://doi.org/10.18653/2026.findings-emnlp.2472
%P 53191-53201

Markdown (Informal)

[Soft Sequence Policy Optimization](https://aclanthology.org/2026.findings-emnlp.2472/) (Glazyrina et al., Findings 2026)

• Soft Sequence Policy Optimization (Glazyrina et al., Findings 2026)

ACL

• Svetlana Glazyrina, Maksim Kryzhanovskiy, and Roman Ischenko. 2026. Soft Sequence Policy Optimization. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 53191–53201, Budapest, Hungary. Association for Computational Linguistics.