<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="leng-etal-2026-groupdpo">
    <titleInfo>
        <title>GroupDPO: Memory-Efficient Group-Wise Direct Preference Optimization</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Jixuan</namePart>
        <namePart type="family">Leng</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Si</namePart>
        <namePart type="family">Si</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Hsiang-Fu</namePart>
        <namePart type="family">Yu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Vinod</namePart>
        <namePart type="family">Raman</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Inderjit</namePart>
        <namePart type="given">S</namePart>
        <namePart type="family">Dhillon</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2026</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Hung-Yi</namePart>
            <namePart type="family">Lee</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sunipa</namePart>
            <namePart type="family">Dev</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jessy</namePart>
            <namePart type="family">Li</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Desmond</namePart>
            <namePart type="family">Elliott</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-462-0</identifier>
    </relatedItem>
    <abstract>Preference optimization is widely used to align Large Language Models (LLMs) with preference feedback. However, most existing methods train on a single positive-negative pair per prompt, discarding additional supervision available in preference datasets that typically contain multiple candidate responses. Motivated by this limitation, recent work explores group-wise preference optimization, which jointly contrasts multiple responses for the same prompt, but its empirical behavior and scalability remain underexplored due to the memory overhead of group-coupled objectives. In this work, we present a unified empirical and systems study of group-wise preference optimization and develop a memory-efficient implementation for group-coupled objectives. By instantiating first-order linearization with objective-specific per-response coefficients, our implementation preserves first-order gradients while decoupling samples during backpropagation, substantially reducing peak memory usage and enabling scalable training with larger groups. Across offline and online settings, we show that leveraging multiple responses consistently outperforms single-pair training. Furthermore, incorporating a negative log-likelihood (NLL) term on positive responses is critical for both performance gains and training stability.</abstract>
    <identifier type="citekey">leng-etal-2026-groupdpo</identifier>
    <location>
        <url>https://aclanthology.org/2026.findings-emnlp.3/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>29</start>
            <end>51</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T GroupDPO: Memory-Efficient Group-Wise Direct Preference Optimization
%A Leng, Jixuan
%A Si, Si
%A Yu, Hsiang-Fu
%A Raman, Vinod
%A Dhillon, Inderjit S.
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Findings of the Association for Computational Linguistics: EMNLP 2026
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-462-0
%F leng-etal-2026-groupdpo
%X Preference optimization is widely used to align Large Language Models (LLMs) with preference feedback. However, most existing methods train on a single positive-negative pair per prompt, discarding additional supervision available in preference datasets that typically contain multiple candidate responses. Motivated by this limitation, recent work explores group-wise preference optimization, which jointly contrasts multiple responses for the same prompt, but its empirical behavior and scalability remain underexplored due to the memory overhead of group-coupled objectives. In this work, we present a unified empirical and systems study of group-wise preference optimization and develop a memory-efficient implementation for group-coupled objectives. By instantiating first-order linearization with objective-specific per-response coefficients, our implementation preserves first-order gradients while decoupling samples during backpropagation, substantially reducing peak memory usage and enabling scalable training with larger groups. Across offline and online settings, we show that leveraging multiple responses consistently outperforms single-pair training. Furthermore, incorporating a negative log-likelihood (NLL) term on positive responses is critical for both performance gains and training stability.
%U https://aclanthology.org/2026.findings-emnlp.3/
%P 29-51

Markdown (Informal)

[GroupDPO: Memory-Efficient Group-Wise Direct Preference Optimization](https://aclanthology.org/2026.findings-emnlp.3/) (Leng et al., Findings 2026)

• GroupDPO: Memory-Efficient Group-Wise Direct Preference Optimization (Leng et al., Findings 2026)

ACL

• Jixuan Leng, Si Si, Hsiang-Fu Yu, Vinod Raman, and Inderjit S Dhillon. 2026. GroupDPO: Memory-Efficient Group-Wise Direct Preference Optimization. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 29–51, Budapest, Hungary. Association for Computational Linguistics.