<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="soubki-rambow-2025-machine">
    <titleInfo>
        <title>Machine Theory of Mind Needs Machine Validation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Adil</namePart>
        <namePart type="family">Soubki</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Owen</namePart>
        <namePart type="family">Rambow</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: ACL 2025</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Wanxiang</namePart>
            <namePart type="family">Che</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Joyce</namePart>
            <namePart type="family">Nabende</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ekaterina</namePart>
            <namePart type="family">Shutova</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Mohammad</namePart>
            <namePart type="given">Taher</namePart>
            <namePart type="family">Pilehvar</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Vienna, Austria</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-256-5</identifier>
    </relatedItem>
    <abstract>In the last couple years, there has been a flood of interest in studying the extent to which language models (LMs) have a theory of mind (ToM) — the ability to ascribe mental states to themselves and others. The results provide an unclear picture of the current state of the art, with some finding near-human performance and others near-zero. To make sense of this landscape, we perform a survey of 16 recent studies aimed at measuring ToM in LMs and find that, while almost all perform checks for human identifiable issues, less than half do so for patterns only a machine might exploit. Among those that do perform such validation, which we call machine validation, none identify LMs to exceed human performance. We conclude that the datasets that show high LM performance on ToM tasks are easier than their peers, likely due to the presence of spurious patterns in the data, and we caution against building ToM benchmarks relying solely on human validation of the data.</abstract>
    <identifier type="citekey">soubki-rambow-2025-machine</identifier>
    <identifier type="doi">10.18653/v1/2025.findings-acl.951</identifier>
    <location>
        <url>https://aclanthology.org/2025.findings-acl.951/</url>
    </location>
    <part>
        <date>2025-07</date>
        <extent unit="page">
            <start>18495</start>
            <end>18505</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Machine Theory of Mind Needs Machine Validation
%A Soubki, Adil
%A Rambow, Owen
%Y Che, Wanxiang
%Y Nabende, Joyce
%Y Shutova, Ekaterina
%Y Pilehvar, Mohammad Taher
%S Findings of the Association for Computational Linguistics: ACL 2025
%D 2025
%8 July
%I Association for Computational Linguistics
%C Vienna, Austria
%@ 979-8-89176-256-5
%F soubki-rambow-2025-machine
%X In the last couple years, there has been a flood of interest in studying the extent to which language models (LMs) have a theory of mind (ToM) — the ability to ascribe mental states to themselves and others. The results provide an unclear picture of the current state of the art, with some finding near-human performance and others near-zero. To make sense of this landscape, we perform a survey of 16 recent studies aimed at measuring ToM in LMs and find that, while almost all perform checks for human identifiable issues, less than half do so for patterns only a machine might exploit. Among those that do perform such validation, which we call machine validation, none identify LMs to exceed human performance. We conclude that the datasets that show high LM performance on ToM tasks are easier than their peers, likely due to the presence of spurious patterns in the data, and we caution against building ToM benchmarks relying solely on human validation of the data.
%R 10.18653/v1/2025.findings-acl.951
%U https://aclanthology.org/2025.findings-acl.951/
%U https://doi.org/10.18653/v1/2025.findings-acl.951
%P 18495-18505

Markdown (Informal)

[Machine Theory of Mind Needs Machine Validation](https://aclanthology.org/2025.findings-acl.951/) (Soubki & Rambow, Findings 2025)

• Machine Theory of Mind Needs Machine Validation (Soubki & Rambow, Findings 2025)

ACL

• Adil Soubki and Owen Rambow. 2025. Machine Theory of Mind Needs Machine Validation. In Findings of the Association for Computational Linguistics: ACL 2025, pages 18495–18505, Vienna, Austria. Association for Computational Linguistics.