<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="seals-shalin-2024-evaluating">
    <titleInfo>
        <title>Evaluating the Deductive Competence of Large Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">S</namePart>
        <namePart type="family">Seals</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Valerie</namePart>
        <namePart type="family">Shalin</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-06</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Kevin</namePart>
            <namePart type="family">Duh</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Helena</namePart>
            <namePart type="family">Gomez</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Steven</namePart>
            <namePart type="family">Bethard</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Mexico City, Mexico</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>The development of highly fluent large language models (LLMs) has prompted increased interest in assessing their reasoning and problem-solving capabilities. We investigate whether several LLMs can solve a classic type of deductive reasoning problem from the cognitive science literature. The tested LLMs have limited abilities to solve these problems in their conventional form. We performed follow up experiments to investigate if changes to the presentation format and content improve model performance. We do find performance differences between conditions; however, they do not improve overall performance. Moreover, we find that performance interacts with presentation format and content in unexpected ways that differ from human performance. Overall, our results suggest that LLMs have unique reasoning biases that are only partially predicted from human reasoning performance and the human-generated language corpora that informs them.</abstract>
    <identifier type="citekey">seals-shalin-2024-evaluating</identifier>
    <identifier type="doi">10.18653/v1/2024.naacl-long.476</identifier>
    <location>
        <url>https://aclanthology.org/2024.naacl-long.476</url>
    </location>
    <part>
        <date>2024-06</date>
        <extent unit="page">
            <start>8614</start>
            <end>8630</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Evaluating the Deductive Competence of Large Language Models
%A Seals, S.
%A Shalin, Valerie
%Y Duh, Kevin
%Y Gomez, Helena
%Y Bethard, Steven
%S Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
%D 2024
%8 June
%I Association for Computational Linguistics
%C Mexico City, Mexico
%F seals-shalin-2024-evaluating
%X The development of highly fluent large language models (LLMs) has prompted increased interest in assessing their reasoning and problem-solving capabilities. We investigate whether several LLMs can solve a classic type of deductive reasoning problem from the cognitive science literature. The tested LLMs have limited abilities to solve these problems in their conventional form. We performed follow up experiments to investigate if changes to the presentation format and content improve model performance. We do find performance differences between conditions; however, they do not improve overall performance. Moreover, we find that performance interacts with presentation format and content in unexpected ways that differ from human performance. Overall, our results suggest that LLMs have unique reasoning biases that are only partially predicted from human reasoning performance and the human-generated language corpora that informs them.
%R 10.18653/v1/2024.naacl-long.476
%U https://aclanthology.org/2024.naacl-long.476
%U https://doi.org/10.18653/v1/2024.naacl-long.476
%P 8614-8630

Markdown (Informal)

[Evaluating the Deductive Competence of Large Language Models](https://aclanthology.org/2024.naacl-long.476) (Seals & Shalin, NAACL 2024)

• Evaluating the Deductive Competence of Large Language Models (Seals & Shalin, NAACL 2024)

ACL

• S Seals and Valerie Shalin. 2024. Evaluating the Deductive Competence of Large Language Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8614–8630, Mexico City, Mexico. Association for Computational Linguistics.