<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="shaik-etal-2025-mismatched">
    <titleInfo>
        <title>A MISMATCHED Benchmark for Scientific Natural Language Inference</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Firoz</namePart>
        <namePart type="family">Shaik</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Mobashir</namePart>
        <namePart type="family">Sadat</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Nikita</namePart>
        <namePart type="family">Gautam</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Doina</namePart>
        <namePart type="family">Caragea</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Cornelia</namePart>
        <namePart type="family">Caragea</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: ACL 2025</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Wanxiang</namePart>
            <namePart type="family">Che</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Joyce</namePart>
            <namePart type="family">Nabende</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ekaterina</namePart>
            <namePart type="family">Shutova</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Mohammad</namePart>
            <namePart type="given">Taher</namePart>
            <namePart type="family">Pilehvar</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Vienna, Austria</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-256-5</identifier>
    </relatedItem>
    <abstract>Scientific Natural Language Inference (NLI) is the task of predicting the semantic relation between a pair of sentences extracted from research articles. Existing datasets for this task are derived from various computer science (CS) domains, whereas non-CS domains are completely ignored. In this paper, we introduce a novel evaluation benchmark for scientific NLI, called MisMatched. The new MisMatched benchmark covers three non-CS domains–Psychology, Engineering, and Public Health, and contains 2,700 human annotated sentence pairs. We establish strong baselines on MisMatched using both Pre-trained Small Language Models (SLMs) and Large Language Models (LLMs). Our best performing baseline shows a Macro F1 of only 78.17% illustrating the substantial headroom for future improvements. In addition to introducing the MisMatched benchmark, we show that incorporating sentence pairs having an implicit scientific NLI relation between them in model training improves their performance on scientific NLI. We make our dataset and code publicly available on GitHub.</abstract>
    <identifier type="citekey">shaik-etal-2025-mismatched</identifier>
    <identifier type="doi">10.18653/v1/2025.findings-acl.1109</identifier>
    <location>
        <url>https://aclanthology.org/2025.findings-acl.1109/</url>
    </location>
    <part>
        <date>2025-07</date>
        <extent unit="page">
            <start>21524</start>
            <end>21538</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T A MISMATCHED Benchmark for Scientific Natural Language Inference
%A Shaik, Firoz
%A Sadat, Mobashir
%A Gautam, Nikita
%A Caragea, Doina
%A Caragea, Cornelia
%Y Che, Wanxiang
%Y Nabende, Joyce
%Y Shutova, Ekaterina
%Y Pilehvar, Mohammad Taher
%S Findings of the Association for Computational Linguistics: ACL 2025
%D 2025
%8 July
%I Association for Computational Linguistics
%C Vienna, Austria
%@ 979-8-89176-256-5
%F shaik-etal-2025-mismatched
%X Scientific Natural Language Inference (NLI) is the task of predicting the semantic relation between a pair of sentences extracted from research articles. Existing datasets for this task are derived from various computer science (CS) domains, whereas non-CS domains are completely ignored. In this paper, we introduce a novel evaluation benchmark for scientific NLI, called MisMatched. The new MisMatched benchmark covers three non-CS domains–Psychology, Engineering, and Public Health, and contains 2,700 human annotated sentence pairs. We establish strong baselines on MisMatched using both Pre-trained Small Language Models (SLMs) and Large Language Models (LLMs). Our best performing baseline shows a Macro F1 of only 78.17% illustrating the substantial headroom for future improvements. In addition to introducing the MisMatched benchmark, we show that incorporating sentence pairs having an implicit scientific NLI relation between them in model training improves their performance on scientific NLI. We make our dataset and code publicly available on GitHub.
%R 10.18653/v1/2025.findings-acl.1109
%U https://aclanthology.org/2025.findings-acl.1109/
%U https://doi.org/10.18653/v1/2025.findings-acl.1109
%P 21524-21538

Markdown (Informal)

[A MISMATCHED Benchmark for Scientific Natural Language Inference](https://aclanthology.org/2025.findings-acl.1109/) (Shaik et al., Findings 2025)

• A MISMATCHED Benchmark for Scientific Natural Language Inference (Shaik et al., Findings 2025)

ACL

• Firoz Shaik, Mobashir Sadat, Nikita Gautam, Doina Caragea, and Cornelia Caragea. 2025. A MISMATCHED Benchmark for Scientific Natural Language Inference. In Findings of the Association for Computational Linguistics: ACL 2025, pages 21524–21538, Vienna, Austria. Association for Computational Linguistics.