<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="fu-etal-2025-chengyu">
    <titleInfo>
        <title>CHENGYU-BENCH: Benchmarking Large Language Models for Chinese Idiom Understanding and Use</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Yicheng</namePart>
        <namePart type="family">Fu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Zhemin</namePart>
        <namePart type="family">Huang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Liuxin</namePart>
        <namePart type="family">Yang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yumeng</namePart>
        <namePart type="family">Lu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Zhongdongming</namePart>
        <namePart type="family">Dai</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Christos</namePart>
            <namePart type="family">Christodoulopoulos</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Tanmoy</namePart>
            <namePart type="family">Chakraborty</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Carolyn</namePart>
            <namePart type="family">Rose</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Violet</namePart>
            <namePart type="family">Peng</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Suzhou, China</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-332-6</identifier>
    </relatedItem>
    <abstract>Chinese idioms (成语, Chengyu) are concise four-character expressions steeped in history and culture, whose literal translations often fail to capture their full meaning. This complexity makes them challenging for language models to interpret and use correctly. Existing benchmarks focus on narrow tasks—multiple-choice cloze tests, isolated translation, or simple paraphrasing. We introduce CHENGYU-BENCH, a comprehensive benchmark featuring three tasks: (1) Evaluative Connotation, classifying idioms as positive or negative; (2) Appropriateness, detecting incorrect idiom usage in context; and (3) Open Cloze, filling blanks in longer passages without options. CHENGYU-BENCH comprises 2,937 human-verified examples covering 1,765 common idioms sourced from diverse corpora. We evaluate leading LLMs and find they achieve over 95% accuracy on Evaluative Connotation, but only ~85% on Appropriateness and ~40% top-1 accuracy in Open Cloze. Error analysis reveals that most mistakes arise from fundamental misunderstandings of idiom meanings. CHENGYU-BENCH demonstrates that while LLMs can reliably gauge idiom sentiment, they still struggle to grasp the cultural and contextual nuances essential for proper usage. The benchmark and code will be released upon paper acceptance.</abstract>
    <identifier type="citekey">fu-etal-2025-chengyu</identifier>
    <identifier type="doi">10.18653/v1/2025.emnlp-main.119</identifier>
    <location>
        <url>https://aclanthology.org/2025.emnlp-main.119/</url>
    </location>
    <part>
        <date>2025-11</date>
        <extent unit="page">
            <start>2355</start>
            <end>2366</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T CHENGYU-BENCH: Benchmarking Large Language Models for Chinese Idiom Understanding and Use
%A Fu, Yicheng
%A Huang, Zhemin
%A Yang, Liuxin
%A Lu, Yumeng
%A Dai, Zhongdongming
%Y Christodoulopoulos, Christos
%Y Chakraborty, Tanmoy
%Y Rose, Carolyn
%Y Peng, Violet
%S Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
%D 2025
%8 November
%I Association for Computational Linguistics
%C Suzhou, China
%@ 979-8-89176-332-6
%F fu-etal-2025-chengyu
%X Chinese idioms (成语, Chengyu) are concise four-character expressions steeped in history and culture, whose literal translations often fail to capture their full meaning. This complexity makes them challenging for language models to interpret and use correctly. Existing benchmarks focus on narrow tasks—multiple-choice cloze tests, isolated translation, or simple paraphrasing. We introduce CHENGYU-BENCH, a comprehensive benchmark featuring three tasks: (1) Evaluative Connotation, classifying idioms as positive or negative; (2) Appropriateness, detecting incorrect idiom usage in context; and (3) Open Cloze, filling blanks in longer passages without options. CHENGYU-BENCH comprises 2,937 human-verified examples covering 1,765 common idioms sourced from diverse corpora. We evaluate leading LLMs and find they achieve over 95% accuracy on Evaluative Connotation, but only ~85% on Appropriateness and ~40% top-1 accuracy in Open Cloze. Error analysis reveals that most mistakes arise from fundamental misunderstandings of idiom meanings. CHENGYU-BENCH demonstrates that while LLMs can reliably gauge idiom sentiment, they still struggle to grasp the cultural and contextual nuances essential for proper usage. The benchmark and code will be released upon paper acceptance.
%R 10.18653/v1/2025.emnlp-main.119
%U https://aclanthology.org/2025.emnlp-main.119/
%U https://doi.org/10.18653/v1/2025.emnlp-main.119
%P 2355-2366

Markdown (Informal)

[CHENGYU-BENCH: Benchmarking Large Language Models for Chinese Idiom Understanding and Use](https://aclanthology.org/2025.emnlp-main.119/) (Fu et al., EMNLP 2025)

• CHENGYU-BENCH: Benchmarking Large Language Models for Chinese Idiom Understanding and Use (Fu et al., EMNLP 2025)

ACL

• Yicheng Fu, Zhemin Huang, Liuxin Yang, Yumeng Lu, and Zhongdongming Dai. 2025. CHENGYU-BENCH: Benchmarking Large Language Models for Chinese Idiom Understanding and Use. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 2355–2366, Suzhou, China. Association for Computational Linguistics.