<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="eigler-etal-2026-llm">
    <titleInfo>
        <title>LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Lukáš</namePart>
        <namePart type="family">Eigler</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jindřich</namePart>
        <namePart type="family">Libovický</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">David</namePart>
        <namePart type="family">Hurych</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Santosh</namePart>
            <namePart type="family">T.Y.S.S.</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="given">Diego</namePart>
            <namePart type="family">Rodriguez</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ona</namePart>
            <namePart type="family">de Gibert</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">San Diego, California, United States</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-393-7</identifier>
    </relatedItem>
    <abstract>Validating evaluation metrics for NLG typically relies on expensive and time-consuming human annotations, which predominantly exists only for English datasets. We propose LLM as a Meta-Judge, a scalable framework that utilizes LLMs to generate synthetic evaluation datasets via controlled semantic degradation of real data, replacing human judgment. We validate our approach using meta-correlation, measuring the alignment between metric rankings derived from synthetic data and those from standard human benchmarks. Experiments across Machine Translation, Question Answering, and Summarization demonstrate that synthetic validation serves as a reliable proxy for human judgment, achieving meta-correlations exceeding 0.9 in multilingual QA and proves to be a viable alternative where human judgments are unavailable or too expensive to obtain. Our code and data are publicly available at https://github.com/eiglerl/meta-judge.</abstract>
    <identifier type="citekey">eigler-etal-2026-llm</identifier>
    <location>
        <url>https://aclanthology.org/2026.acl-srw.125/</url>
    </location>
    <part>
        <date>2026-07</date>
        <extent unit="page">
            <start>1417</start>
            <end>1435</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation
%A Eigler, Lukáš
%A Libovický, Jindřich
%A Hurych, David
%Y T.Y.S.S., Santosh
%Y Rodriguez, Juan Diego
%Y de Gibert, Ona
%S Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026)
%D 2026
%8 July
%I Association for Computational Linguistics
%C San Diego, California, United States
%@ 979-8-89176-393-7
%F eigler-etal-2026-llm
%X Validating evaluation metrics for NLG typically relies on expensive and time-consuming human annotations, which predominantly exists only for English datasets. We propose LLM as a Meta-Judge, a scalable framework that utilizes LLMs to generate synthetic evaluation datasets via controlled semantic degradation of real data, replacing human judgment. We validate our approach using meta-correlation, measuring the alignment between metric rankings derived from synthetic data and those from standard human benchmarks. Experiments across Machine Translation, Question Answering, and Summarization demonstrate that synthetic validation serves as a reliable proxy for human judgment, achieving meta-correlations exceeding 0.9 in multilingual QA and proves to be a viable alternative where human judgments are unavailable or too expensive to obtain. Our code and data are publicly available at https://github.com/eiglerl/meta-judge.
%U https://aclanthology.org/2026.acl-srw.125/
%P 1417-1435

Markdown (Informal)

[LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation](https://aclanthology.org/2026.acl-srw.125/) (Eigler et al., ACL 2026)

• LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation (Eigler et al., ACL 2026)

ACL

• Lukáš Eigler, Jindřich Libovický, and David Hurych. 2026. LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026), pages 1417–1435, San Diego, California, United States. Association for Computational Linguistics.