<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="gevers-daelemans-2026-get">
    <titleInfo>
        <title>Do You Get the Hint? Benchmarking LLMs on the Board Game Concept</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Ine</namePart>
        <namePart type="family">Gevers</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Walter</namePart>
        <namePart type="family">Daelemans</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: ACL 2026</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Maria</namePart>
            <namePart type="family">Liakata</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Viviane</namePart>
            <namePart type="given">P</namePart>
            <namePart type="family">Moreira</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jiajun</namePart>
            <namePart type="family">Zhang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">David</namePart>
            <namePart type="family">Jurgens</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">San Diego, California, United States</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-395-1</identifier>
    </relatedItem>
    <abstract>Large language models (LLMs) have achieved striking successes on many benchmarks, yet recent studies continue to expose fundamental weaknesses. In this paper, we introduce Concept, a simple word-guessing board game, as a benchmark for probing abductive reasoning. Our results show that this game, easily solved by humans (with a success rate of over 90%), is still very challenging for state-of-the-art LLMs (no model exceeds 40% success rate). Specifically, we observe that LLMs struggle with interpreting other players’ strategic intents, and with correcting initial hypotheses given sequential information updates. In addition, we extend the evaluation across multiple languages, and find that the LLM performance drops further in lower-resource languages (Dutch, French, and Spanish) compared to English.</abstract>
    <identifier type="citekey">gevers-daelemans-2026-get</identifier>
    <location>
        <url>https://aclanthology.org/2026.findings-acl.1219/</url>
    </location>
    <part>
        <date>2026-07</date>
        <extent unit="page">
            <start>24358</start>
            <end>24371</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Do You Get the Hint? Benchmarking LLMs on the Board Game Concept
%A Gevers, Ine
%A Daelemans, Walter
%Y Liakata, Maria
%Y Moreira, Viviane P.
%Y Zhang, Jiajun
%Y Jurgens, David
%S Findings of the Association for Computational Linguistics: ACL 2026
%D 2026
%8 July
%I Association for Computational Linguistics
%C San Diego, California, United States
%@ 979-8-89176-395-1
%F gevers-daelemans-2026-get
%X Large language models (LLMs) have achieved striking successes on many benchmarks, yet recent studies continue to expose fundamental weaknesses. In this paper, we introduce Concept, a simple word-guessing board game, as a benchmark for probing abductive reasoning. Our results show that this game, easily solved by humans (with a success rate of over 90%), is still very challenging for state-of-the-art LLMs (no model exceeds 40% success rate). Specifically, we observe that LLMs struggle with interpreting other players’ strategic intents, and with correcting initial hypotheses given sequential information updates. In addition, we extend the evaluation across multiple languages, and find that the LLM performance drops further in lower-resource languages (Dutch, French, and Spanish) compared to English.
%U https://aclanthology.org/2026.findings-acl.1219/
%P 24358-24371

Markdown (Informal)

[Do You Get the Hint? Benchmarking LLMs on the Board Game Concept](https://aclanthology.org/2026.findings-acl.1219/) (Gevers & Daelemans, Findings 2026)

• Do You Get the Hint? Benchmarking LLMs on the Board Game Concept (Gevers & Daelemans, Findings 2026)

ACL

• Ine Gevers and Walter Daelemans. 2026. Do You Get the Hint? Benchmarking LLMs on the Board Game Concept. In Findings of the Association for Computational Linguistics: ACL 2026, pages 24358–24371, San Diego, California, United States. Association for Computational Linguistics.