<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="huang-etal-2023-reduce">
    <titleInfo>
        <title>Reduce Human Labor On Evaluating Conversational Information Retrieval System: A Human-Machine Collaboration Approach</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Chen</namePart>
        <namePart type="family">Huang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Peixin</namePart>
        <namePart type="family">Qin</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Wenqiang</namePart>
        <namePart type="family">Lei</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jiancheng</namePart>
        <namePart type="family">Lv</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Evaluating conversational information retrieval (CIR) systems is a challenging task that requires a significant amount of human labor for annotation. It is imperative to invest significant effort into researching more labor-effective methods for evaluating CIR systems. To touch upon this challenge, we take the first step to involve active testing in CIR evaluation and propose a novel method, called HomCoE. It strategically selects a few data for human annotation, then calibrates the evaluation results to eliminate evaluation biases. As such, it makes an accurate evaluation of the CIR system at low human labor. We experimentally reveal that it consumes less than 1% of human labor and achieves a consistency rate of 95%-99% with human evaluation results. This emphasizes the superiority of our method over other baselines.</abstract>
    <identifier type="citekey">huang-etal-2023-reduce</identifier>
    <identifier type="doi">10.18653/v1/2023.emnlp-main.670</identifier>
    <location>
        <url>https://aclanthology.org/2023.emnlp-main.670</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>10876</start>
            <end>10891</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Reduce Human Labor On Evaluating Conversational Information Retrieval System: A Human-Machine Collaboration Approach
%A Huang, Chen
%A Qin, Peixin
%A Lei, Wenqiang
%A Lv, Jiancheng
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F huang-etal-2023-reduce
%X Evaluating conversational information retrieval (CIR) systems is a challenging task that requires a significant amount of human labor for annotation. It is imperative to invest significant effort into researching more labor-effective methods for evaluating CIR systems. To touch upon this challenge, we take the first step to involve active testing in CIR evaluation and propose a novel method, called HomCoE. It strategically selects a few data for human annotation, then calibrates the evaluation results to eliminate evaluation biases. As such, it makes an accurate evaluation of the CIR system at low human labor. We experimentally reveal that it consumes less than 1% of human labor and achieves a consistency rate of 95%-99% with human evaluation results. This emphasizes the superiority of our method over other baselines.
%R 10.18653/v1/2023.emnlp-main.670
%U https://aclanthology.org/2023.emnlp-main.670
%U https://doi.org/10.18653/v1/2023.emnlp-main.670
%P 10876-10891

Markdown (Informal)

[Reduce Human Labor On Evaluating Conversational Information Retrieval System: A Human-Machine Collaboration Approach](https://aclanthology.org/2023.emnlp-main.670) (Huang et al., EMNLP 2023)

• Reduce Human Labor On Evaluating Conversational Information Retrieval System: A Human-Machine Collaboration Approach (Huang et al., EMNLP 2023)

ACL

• Chen Huang, Peixin Qin, Wenqiang Lei, and Jiancheng Lv. 2023. Reduce Human Labor On Evaluating Conversational Information Retrieval System: A Human-Machine Collaboration Approach. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10876–10891, Singapore. Association for Computational Linguistics.