<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="cheng-etal-2026-multi">
    <titleInfo>
        <title>Multi-Expert Conformal Risk Control for Pairwise LLM Judging in Open-Ended Dialogue</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Ming</namePart>
        <namePart type="family">Cheng</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yusheng</namePart>
        <namePart type="family">Dai</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Qiuhong</namePart>
        <namePart type="family">Ke</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Zhaolin</namePart>
        <namePart type="family">Chen</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Lizhen</namePart>
        <namePart type="family">Qu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Hung-Yi</namePart>
            <namePart type="family">Lee</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sunipa</namePart>
            <namePart type="family">Dev</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jessy</namePart>
            <namePart type="family">Li</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Desmond</namePart>
            <namePart type="family">Elliott</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-459-0</identifier>
    </relatedItem>
    <abstract>In this paper, we explore multi-expert Conformal Risk Control (CRC) algorithms for pairwise LLM-as-a-Judge evaluation in open-ended dialogue. Our core insight is that multi-expert aggregation offers a complementary remedy to CRC: whereas CRC controls risk at the decision threshold through abstention, aggregation sanitizes the scoring function at its source. Guided by this, we first design two multi-expert CRC methods: Score Averaging and Decision Voting, which aggregate at the score and decision levels, respectively. While both strategies outperform single-expert methods on homogeneous expert panels, on heterogeneous LLM judges they remain risk-valid but recover only limited coverage, because a uniform threshold cannot match the experts’ distinct scoring scales. To resolve this issue, we further propose Marginal-Calibrated Conformal Consensus (MC³): it captures distinct per-expert scales via initial threshold ratios, while jointly tuning a unified decision function C\ensuremath_t(x) applied identically in both calibration and test, thereby preserving exchangeability. To evaluate our framework, we construct Panel, a 1,800-pair human pairwise-preference benchmark for open-ended dialogue. It is built on responses generated by four open-weight LLMs over dialogue contexts from three domains (ESConv, MSC, DREAM), with full logit access. In experiments, we find that both Score Averaging and Decision Voting substantially improve accuracy and acceptance rate on homogeneous panels. Notably, MC³ extends these gains to heterogeneous panels by accommodating distinct per-expert scoring scales across all three datasets.</abstract>
    <identifier type="citekey">cheng-etal-2026-multi</identifier>
    <location>
        <url>https://aclanthology.org/2026.emnlp-main.53/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>1201</start>
            <end>1215</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Multi-Expert Conformal Risk Control for Pairwise LLM Judging in Open-Ended Dialogue
%A Cheng, Ming
%A Dai, Yusheng
%A Ke, Qiuhong
%A Chen, Zhaolin
%A Qu, Lizhen
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-459-0
%F cheng-etal-2026-multi
%X In this paper, we explore multi-expert Conformal Risk Control (CRC) algorithms for pairwise LLM-as-a-Judge evaluation in open-ended dialogue. Our core insight is that multi-expert aggregation offers a complementary remedy to CRC: whereas CRC controls risk at the decision threshold through abstention, aggregation sanitizes the scoring function at its source. Guided by this, we first design two multi-expert CRC methods: Score Averaging and Decision Voting, which aggregate at the score and decision levels, respectively. While both strategies outperform single-expert methods on homogeneous expert panels, on heterogeneous LLM judges they remain risk-valid but recover only limited coverage, because a uniform threshold cannot match the experts’ distinct scoring scales. To resolve this issue, we further propose Marginal-Calibrated Conformal Consensus (MC³): it captures distinct per-expert scales via initial threshold ratios, while jointly tuning a unified decision function C\ensuremath_t(x) applied identically in both calibration and test, thereby preserving exchangeability. To evaluate our framework, we construct Panel, a 1,800-pair human pairwise-preference benchmark for open-ended dialogue. It is built on responses generated by four open-weight LLMs over dialogue contexts from three domains (ESConv, MSC, DREAM), with full logit access. In experiments, we find that both Score Averaging and Decision Voting substantially improve accuracy and acceptance rate on homogeneous panels. Notably, MC³ extends these gains to heterogeneous panels by accommodating distinct per-expert scoring scales across all three datasets.
%U https://aclanthology.org/2026.emnlp-main.53/
%P 1201-1215

Markdown (Informal)

[Multi-Expert Conformal Risk Control for Pairwise LLM Judging in Open-Ended Dialogue](https://aclanthology.org/2026.emnlp-main.53/) (Cheng et al., EMNLP 2026)

• Multi-Expert Conformal Risk Control for Pairwise LLM Judging in Open-Ended Dialogue (Cheng et al., EMNLP 2026)

ACL

• Ming Cheng, Yusheng Dai, Qiuhong Ke, Zhaolin Chen, and Lizhen Qu. 2026. Multi-Expert Conformal Risk Control for Pairwise LLM Judging in Open-Ended Dialogue. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 1201–1215, Budapest, Hungary. Association for Computational Linguistics.