<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="sileo-lernould-2023-mindgames">
    <titleInfo>
        <title>MindGames: Targeting Theory of Mind in Large Language Models with Dynamic Epistemic Modal Logic</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Damien</namePart>
        <namePart type="family">Sileo</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Antoine</namePart>
        <namePart type="family">Lernould</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2023</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Theory of Mind (ToM) is a critical component of intelligence but its assessment remains the subject of heated debates. Prior research applied human ToM assessments to natural language processing models using either human-created standardized tests or rule-based templates. However, these methods primarily focus on simplistic reasoning and require further validation. Here, we leverage dynamic epistemic logic to isolate a particular component of ToM and to generate controlled problems. We also introduce new verbalization techniques to express these problems in English natural language. Our findings indicate that some language model scaling (from 70M to 6B and 350M to 174B) does not consistently yield results better than random chance. While GPT-4 demonstrates superior epistemic reasoning capabilities, there is still room for improvement. Our code and datasets are publicly available.</abstract>
    <identifier type="citekey">sileo-lernould-2023-mindgames</identifier>
    <identifier type="doi">10.18653/v1/2023.findings-emnlp.303</identifier>
    <location>
        <url>https://aclanthology.org/2023.findings-emnlp.303</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>4570</start>
            <end>4577</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T MindGames: Targeting Theory of Mind in Large Language Models with Dynamic Epistemic Modal Logic
%A Sileo, Damien
%A Lernould, Antoine
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Findings of the Association for Computational Linguistics: EMNLP 2023
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F sileo-lernould-2023-mindgames
%X Theory of Mind (ToM) is a critical component of intelligence but its assessment remains the subject of heated debates. Prior research applied human ToM assessments to natural language processing models using either human-created standardized tests or rule-based templates. However, these methods primarily focus on simplistic reasoning and require further validation. Here, we leverage dynamic epistemic logic to isolate a particular component of ToM and to generate controlled problems. We also introduce new verbalization techniques to express these problems in English natural language. Our findings indicate that some language model scaling (from 70M to 6B and 350M to 174B) does not consistently yield results better than random chance. While GPT-4 demonstrates superior epistemic reasoning capabilities, there is still room for improvement. Our code and datasets are publicly available.
%R 10.18653/v1/2023.findings-emnlp.303
%U https://aclanthology.org/2023.findings-emnlp.303
%U https://doi.org/10.18653/v1/2023.findings-emnlp.303
%P 4570-4577

Markdown (Informal)

[MindGames: Targeting Theory of Mind in Large Language Models with Dynamic Epistemic Modal Logic](https://aclanthology.org/2023.findings-emnlp.303) (Sileo & Lernould, Findings 2023)

• MindGames: Targeting Theory of Mind in Large Language Models with Dynamic Epistemic Modal Logic (Sileo & Lernould, Findings 2023)

ACL

• Damien Sileo and Antoine Lernould. 2023. MindGames: Targeting Theory of Mind in Large Language Models with Dynamic Epistemic Modal Logic. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4570–4577, Singapore. Association for Computational Linguistics.