<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="edman-etal-2025-execute">
    <titleInfo>
        <title>EXECUTE: A Multilingual Benchmark for LLM Token Understanding</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Lukas</namePart>
        <namePart type="family">Edman</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Helmut</namePart>
        <namePart type="family">Schmid</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Alexander</namePart>
        <namePart type="family">Fraser</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: ACL 2025</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Wanxiang</namePart>
            <namePart type="family">Che</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Joyce</namePart>
            <namePart type="family">Nabende</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ekaterina</namePart>
            <namePart type="family">Shutova</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Mohammad</namePart>
            <namePart type="given">Taher</namePart>
            <namePart type="family">Pilehvar</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Vienna, Austria</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-256-5</identifier>
    </relatedItem>
    <abstract>The CUTE benchmark showed that LLMs struggle with character understanding in English. We extend it to more languages with diverse scripts and writing systems, introducing EXECUTE. Our simplified framework allows easy expansion to any language. Tests across multiple LLMs reveal that challenges in other languages are not always on the character level as in English. Some languages show word-level processing issues, some show no issues at all. We also examine sub-character tasks in Chinese, Japanese, and Korean to assess LLMs’ understanding of character components.</abstract>
    <identifier type="citekey">edman-etal-2025-execute</identifier>
    <identifier type="doi">10.18653/v1/2025.findings-acl.95</identifier>
    <location>
        <url>https://aclanthology.org/2025.findings-acl.95/</url>
    </location>
    <part>
        <date>2025-07</date>
        <extent unit="page">
            <start>1878</start>
            <end>1887</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T EXECUTE: A Multilingual Benchmark for LLM Token Understanding
%A Edman, Lukas
%A Schmid, Helmut
%A Fraser, Alexander
%Y Che, Wanxiang
%Y Nabende, Joyce
%Y Shutova, Ekaterina
%Y Pilehvar, Mohammad Taher
%S Findings of the Association for Computational Linguistics: ACL 2025
%D 2025
%8 July
%I Association for Computational Linguistics
%C Vienna, Austria
%@ 979-8-89176-256-5
%F edman-etal-2025-execute
%X The CUTE benchmark showed that LLMs struggle with character understanding in English. We extend it to more languages with diverse scripts and writing systems, introducing EXECUTE. Our simplified framework allows easy expansion to any language. Tests across multiple LLMs reveal that challenges in other languages are not always on the character level as in English. Some languages show word-level processing issues, some show no issues at all. We also examine sub-character tasks in Chinese, Japanese, and Korean to assess LLMs’ understanding of character components.
%R 10.18653/v1/2025.findings-acl.95
%U https://aclanthology.org/2025.findings-acl.95/
%U https://doi.org/10.18653/v1/2025.findings-acl.95
%P 1878-1887

Markdown (Informal)

[EXECUTE: A Multilingual Benchmark for LLM Token Understanding](https://aclanthology.org/2025.findings-acl.95/) (Edman et al., Findings 2025)

• EXECUTE: A Multilingual Benchmark for LLM Token Understanding (Edman et al., Findings 2025)

ACL

• Lukas Edman, Helmut Schmid, and Alexander Fraser. 2025. EXECUTE: A Multilingual Benchmark for LLM Token Understanding. In Findings of the Association for Computational Linguistics: ACL 2025, pages 1878–1887, Vienna, Austria. Association for Computational Linguistics.