<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="schopf-farber-2026-idea">
    <titleInfo>
        <title>Is This Idea Novel? An Automated Benchmark for Judgment of Research Ideas</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Tim</namePart>
        <namePart type="family">Schopf</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Michael</namePart>
        <namePart type="family">Färber</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-05</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Fifteenth Language Resources and Evaluation Conference</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Stelios</namePart>
            <namePart type="family">Piperidis</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Núria</namePart>
            <namePart type="family">Bel</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Henk</namePart>
            <namePart type="family">van den Heuvel</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Nancy</namePart>
            <namePart type="family">Ide</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Simon</namePart>
            <namePart type="family">Krek</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Antonio</namePart>
            <namePart type="family">Toral</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>ELRA Language Resource Association</publisher>
            <place>
                <placeTerm type="text">Palma de Mallorca, Spain</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Judging the novelty of research ideas is crucial for advancing science, enabling the identification of unexplored directions, and ensuring contributions meaningfully extend existing knowledge rather than reiterate minor variations. However, given the exponential growth of scientific literature, manually judging the novelty of research ideas through literature reviews is labor-intensive, subjective, and infeasible at scale. Therefore, recent efforts have proposed automated approaches for research idea novelty judgment. Yet, evaluation of these approaches remains largely inconsistent and is typically based on non-standardized human evaluations, hindering large-scale, comparable evaluations. To address this, we introduce RINoBench, the first comprehensive benchmark for large-scale evaluation of research idea novelty judgments. It comprises 1,381 research ideas derived from and judged by human experts as well as nine automated evaluation metrics designed to assess both rubric-based novelty scores and textual justifications of novelty judgments. Using this benchmark, we evaluate several state-of-the-art large language models (LLMs) on their ability to judge the novelty of research ideas. Our findings reveal that while LLM-generated reasoning closely mirrors human rationales, this alignment does not reliably translate into accurate novelty judgments, which diverge significantly from human gold standard judgments—even among leading reasoning-capable models. Data and code available at: https://github.com/TimSchopf/RINoBench</abstract>
    <identifier type="citekey">schopf-farber-2026-idea</identifier>
    <identifier type="doi">10.63317/4c3gy3f7epnj</identifier>
    <location>
        <url>https://aclanthology.org/2026.lrec-1.370/</url>
    </location>
    <part>
        <date>2026-05</date>
        <extent unit="page">
            <start>4716</start>
            <end>4727</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Is This Idea Novel? An Automated Benchmark for Judgment of Research Ideas
%A Schopf, Tim
%A Färber, Michael
%Y Piperidis, Stelios
%Y Bel, Núria
%Y van den Heuvel, Henk
%Y Ide, Nancy
%Y Krek, Simon
%Y Toral, Antonio
%S Proceedings of the Fifteenth Language Resources and Evaluation Conference
%D 2026
%8 May
%I ELRA Language Resource Association
%C Palma de Mallorca, Spain
%F schopf-farber-2026-idea
%X Judging the novelty of research ideas is crucial for advancing science, enabling the identification of unexplored directions, and ensuring contributions meaningfully extend existing knowledge rather than reiterate minor variations. However, given the exponential growth of scientific literature, manually judging the novelty of research ideas through literature reviews is labor-intensive, subjective, and infeasible at scale. Therefore, recent efforts have proposed automated approaches for research idea novelty judgment. Yet, evaluation of these approaches remains largely inconsistent and is typically based on non-standardized human evaluations, hindering large-scale, comparable evaluations. To address this, we introduce RINoBench, the first comprehensive benchmark for large-scale evaluation of research idea novelty judgments. It comprises 1,381 research ideas derived from and judged by human experts as well as nine automated evaluation metrics designed to assess both rubric-based novelty scores and textual justifications of novelty judgments. Using this benchmark, we evaluate several state-of-the-art large language models (LLMs) on their ability to judge the novelty of research ideas. Our findings reveal that while LLM-generated reasoning closely mirrors human rationales, this alignment does not reliably translate into accurate novelty judgments, which diverge significantly from human gold standard judgments—even among leading reasoning-capable models. Data and code available at: https://github.com/TimSchopf/RINoBench
%R 10.63317/4c3gy3f7epnj
%U https://aclanthology.org/2026.lrec-1.370/
%U https://doi.org/10.63317/4c3gy3f7epnj
%P 4716-4727

Markdown (Informal)

[Is This Idea Novel? An Automated Benchmark for Judgment of Research Ideas](https://aclanthology.org/2026.lrec-1.370/) (Schopf & Färber, LREC 2026)

• Is This Idea Novel? An Automated Benchmark for Judgment of Research Ideas (Schopf & Färber, LREC 2026)

ACL

• Tim Schopf and Michael Färber. 2026. Is This Idea Novel? An Automated Benchmark for Judgment of Research Ideas. In Proceedings of the Fifteenth Language Resources and Evaluation Conference, pages 4716–4727, Palma de Mallorca, Spain. ELRA Language Resource Association.