<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="weber-etal-2024-donkii">
    <titleInfo>
        <title>Donkii: Characterizing and Detecting Errors in Instruction-Tuning Datasets</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Leon</namePart>
        <namePart type="family">Weber</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Robert</namePart>
        <namePart type="family">Litschko</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ekaterina</namePart>
        <namePart type="family">Artemova</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Barbara</namePart>
        <namePart type="family">Plank</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-03</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of The 18th Linguistic Annotation Workshop (LAW-XVIII)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Sophie</namePart>
            <namePart type="family">Henning</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Manfred</namePart>
            <namePart type="family">Stede</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">St. Julians, Malta</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Instruction tuning has become an integral part of training pipelines for Large Language Models (LLMs) and has been shown to yield strong performance gains. In an orthogonal line of research, Annotation Error Detection (AED) has emerged as a tool for detecting quality problems in gold standard labels. So far, however, the application of AED methods has been limited to classification tasks. It is an open question how well AED methods generalize to language generation settings, which are becoming more widespread via LLMs. In this paper, we present a first and novel benchmark for AED on instruction tuning data: Donkii.It comprises three instruction-tuning datasets enriched with error annotations by experts and semi-automatic methods. We also provide a novel taxonomy of error types for instruction-tuning data.We find that all three datasets contain clear errors, which sometimes propagate directly into instruction-tuned LLMs. We propose four AED baselines for the generative setting and evaluate them extensively on the newly introduced dataset. Our results show that the choice of the right AED method and model size is indeed crucial and derive practical recommendations for how to use AED methods to clean instruction-tuning data.</abstract>
    <identifier type="citekey">weber-etal-2024-donkii</identifier>
    <location>
        <url>https://aclanthology.org/2024.law-1.19</url>
    </location>
    <part>
        <date>2024-03</date>
        <extent unit="page">
            <start>197</start>
            <end>215</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Donkii: Characterizing and Detecting Errors in Instruction-Tuning Datasets
%A Weber, Leon
%A Litschko, Robert
%A Artemova, Ekaterina
%A Plank, Barbara
%Y Henning, Sophie
%Y Stede, Manfred
%S Proceedings of The 18th Linguistic Annotation Workshop (LAW-XVIII)
%D 2024
%8 March
%I Association for Computational Linguistics
%C St. Julians, Malta
%F weber-etal-2024-donkii
%X Instruction tuning has become an integral part of training pipelines for Large Language Models (LLMs) and has been shown to yield strong performance gains. In an orthogonal line of research, Annotation Error Detection (AED) has emerged as a tool for detecting quality problems in gold standard labels. So far, however, the application of AED methods has been limited to classification tasks. It is an open question how well AED methods generalize to language generation settings, which are becoming more widespread via LLMs. In this paper, we present a first and novel benchmark for AED on instruction tuning data: Donkii.It comprises three instruction-tuning datasets enriched with error annotations by experts and semi-automatic methods. We also provide a novel taxonomy of error types for instruction-tuning data.We find that all three datasets contain clear errors, which sometimes propagate directly into instruction-tuned LLMs. We propose four AED baselines for the generative setting and evaluate them extensively on the newly introduced dataset. Our results show that the choice of the right AED method and model size is indeed crucial and derive practical recommendations for how to use AED methods to clean instruction-tuning data.
%U https://aclanthology.org/2024.law-1.19
%P 197-215

Markdown (Informal)

[Donkii: Characterizing and Detecting Errors in Instruction-Tuning Datasets](https://aclanthology.org/2024.law-1.19) (Weber et al., LAW-WS 2024)

• Donkii: Characterizing and Detecting Errors in Instruction-Tuning Datasets (Weber et al., LAW-WS 2024)

ACL

• Leon Weber, Robert Litschko, Ekaterina Artemova, and Barbara Plank. 2024. Donkii: Characterizing and Detecting Errors in Instruction-Tuning Datasets. In Proceedings of The 18th Linguistic Annotation Workshop (LAW-XVIII), pages 197–215, St. Julians, Malta. Association for Computational Linguistics.