<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="devine-2024-sure">
    <titleInfo>
        <title>Are You Sure? Rank Them Again: Repeated Ranking For Better Preference Datasets</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Peter</namePart>
        <namePart type="family">Devine</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Fourth Workshop on Multilingual Representation Learning (MRL 2024)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Jonne</namePart>
            <namePart type="family">Sälevä</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Abraham</namePart>
            <namePart type="family">Owodunni</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Miami, Florida, USA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Training Large Language Models (LLMs) with Reinforcement Learning from AI Feedback (RLAIF) aligns model outputs more closely with human preferences. This involves an evaluator model ranking multiple candidate responses to user prompts. However, the rankings from popular evaluator models such as GPT-4 can be inconsistent.We propose the Repeat Ranking method, in which we evaluate the same responses multiple times and train only on those responses which are consistently ranked. Using 2,714 training prompts in 62 languages, we generated responses from 7 top multilingual LLMs and had GPT-4 rank them five times each. Evaluating on MT-Bench chat benchmarks in six languages, our method outperformed the standard practice of training on all available prompts.Our work highlights the quality versus quantity trade-off in RLAIF dataset generation and offers a stackable strategy for enhancing dataset and thus model quality.</abstract>
    <identifier type="citekey">devine-2024-sure</identifier>
    <location>
        <url>https://aclanthology.org/2024.mrl-1.5</url>
    </location>
    <part>
        <date>2024-11</date>
        <extent unit="page">
            <start>93</start>
            <end>105</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Are You Sure? Rank Them Again: Repeated Ranking For Better Preference Datasets
%A Devine, Peter
%Y Sälevä, Jonne
%Y Owodunni, Abraham
%S Proceedings of the Fourth Workshop on Multilingual Representation Learning (MRL 2024)
%D 2024
%8 November
%I Association for Computational Linguistics
%C Miami, Florida, USA
%F devine-2024-sure
%X Training Large Language Models (LLMs) with Reinforcement Learning from AI Feedback (RLAIF) aligns model outputs more closely with human preferences. This involves an evaluator model ranking multiple candidate responses to user prompts. However, the rankings from popular evaluator models such as GPT-4 can be inconsistent.We propose the Repeat Ranking method, in which we evaluate the same responses multiple times and train only on those responses which are consistently ranked. Using 2,714 training prompts in 62 languages, we generated responses from 7 top multilingual LLMs and had GPT-4 rank them five times each. Evaluating on MT-Bench chat benchmarks in six languages, our method outperformed the standard practice of training on all available prompts.Our work highlights the quality versus quantity trade-off in RLAIF dataset generation and offers a stackable strategy for enhancing dataset and thus model quality.
%U https://aclanthology.org/2024.mrl-1.5
%P 93-105

Markdown (Informal)

[Are You Sure? Rank Them Again: Repeated Ranking For Better Preference Datasets](https://aclanthology.org/2024.mrl-1.5) (Devine, MRL 2024)

• Are You Sure? Rank Them Again: Repeated Ranking For Better Preference Datasets (Devine, MRL 2024)

ACL

• Peter Devine. 2024. Are You Sure? Rank Them Again: Repeated Ranking For Better Preference Datasets. In Proceedings of the Fourth Workshop on Multilingual Representation Learning (MRL 2024), pages 93–105, Miami, Florida, USA. Association for Computational Linguistics.