<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="schopf-etal-2026-think">
    <titleInfo>
        <title>Think-Probe-Respond: Improving Large Language Models as Judges of Research Idea Novelty</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Tim</namePart>
        <namePart type="family">Schopf</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Tobias</namePart>
        <namePart type="family">Schreieder</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Akiko</namePart>
        <namePart type="family">Aizawa</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2026</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Hung-Yi</namePart>
            <namePart type="family">Lee</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sunipa</namePart>
            <namePart type="family">Dev</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jessy</namePart>
            <namePart type="family">Li</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Desmond</namePart>
            <namePart type="family">Elliott</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-462-0</identifier>
    </relatedItem>
    <abstract>Automated novelty judgment can accelerate scientific discovery by enabling efficient evaluation, refinement, and comparison of research ideas. While large language models are increasingly adopted for this task, we investigate a previously overlooked limitation in their judgment capabilities: despite generating reasoning rationales that closely mirror those of human experts, their final novelty judgments often diverge substantially. We demonstrate that this miscalibration stems from a systematic bias towards judging ideas as “medium novel”. To mitigate this, we propose **T**hink-**P**robe-**R**espond (TPR), a lightweight approach that probes latent novelty judgments from hidden states during the reasoning phase and uses the probed judgments to condition the final response. Across strong baselines, TPR improves novelty judgment performance by 22.30% and successfully mitigates the prevalent “medium novelty” bias.</abstract>
    <identifier type="citekey">schopf-etal-2026-think</identifier>
    <location>
        <url>https://aclanthology.org/2026.findings-emnlp.574/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>12119</start>
            <end>12132</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Think-Probe-Respond: Improving Large Language Models as Judges of Research Idea Novelty
%A Schopf, Tim
%A Schreieder, Tobias
%A Aizawa, Akiko
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Findings of the Association for Computational Linguistics: EMNLP 2026
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-462-0
%F schopf-etal-2026-think
%X Automated novelty judgment can accelerate scientific discovery by enabling efficient evaluation, refinement, and comparison of research ideas. While large language models are increasingly adopted for this task, we investigate a previously overlooked limitation in their judgment capabilities: despite generating reasoning rationales that closely mirror those of human experts, their final novelty judgments often diverge substantially. We demonstrate that this miscalibration stems from a systematic bias towards judging ideas as “medium novel”. To mitigate this, we propose **T**hink-**P**robe-**R**espond (TPR), a lightweight approach that probes latent novelty judgments from hidden states during the reasoning phase and uses the probed judgments to condition the final response. Across strong baselines, TPR improves novelty judgment performance by 22.30% and successfully mitigates the prevalent “medium novelty” bias.
%U https://aclanthology.org/2026.findings-emnlp.574/
%P 12119-12132

Markdown (Informal)

[Think-Probe-Respond: Improving Large Language Models as Judges of Research Idea Novelty](https://aclanthology.org/2026.findings-emnlp.574/) (Schopf et al., Findings 2026)

• Think-Probe-Respond: Improving Large Language Models as Judges of Research Idea Novelty (Schopf et al., Findings 2026)

ACL

• Tim Schopf, Tobias Schreieder, and Akiko Aizawa. 2026. Think-Probe-Respond: Improving Large Language Models as Judges of Research Idea Novelty. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 12119–12132, Budapest, Hungary. Association for Computational Linguistics.