<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="gao-etal-2025-consistency">
    <titleInfo>
        <title>Consistency Rating of Semantic Transparency: an Evaluation Method for Metaphor Competence in Idiom Understanding Tasks</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Hui</namePart>
        <namePart type="family">Gao</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jing</namePart>
        <namePart type="family">Zhang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Peng</namePart>
        <namePart type="family">Zhang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Chang</namePart>
        <namePart type="family">Yang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025-01</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 31st International Conference on Computational Linguistics</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Owen</namePart>
            <namePart type="family">Rambow</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Leo</namePart>
            <namePart type="family">Wanner</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Marianna</namePart>
            <namePart type="family">Apidianaki</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Hend</namePart>
            <namePart type="family">Al-Khalifa</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Barbara</namePart>
            <namePart type="given">Di</namePart>
            <namePart type="family">Eugenio</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Steven</namePart>
            <namePart type="family">Schockaert</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Abu Dhabi, UAE</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Idioms condense complex semantics into fixed phrases, and their meaning is often not directly connected to the literal meaning of their constituent words, making idiom comprehension a test of metaphor competence. Metaphor, as a cognitive process in human beings, has not yet found an effective evaluation method to assess the metaphor competence of LLMs (Large Language Models). In this paper, we propose a method to evaluate the metaphor competence of LLMs for the idiom understanding task: the Consistency Rating of Semantic Transparency (CR-ST). This strategy assesses the difficulty of understanding idioms through two dimensions: overall semantic transparency and constituent semantic transparency, aiming to gauge LLMs’ mastery of metaphor competence. Subsequently, we introduce a prompt mechanism-Paraphrase Augmentation Strategy with Self-checking (PASS), based on human language logic, which guides the model to enhance its metaphor competence by explicitly generating idiom paraphrases. We conducted a baseline evaluation of seven LLMs on the CINLID and ChID datasets and analyzed the effectiveness of PASS on different subsets of semantic transparency. The experimental results demonstrate that LLMs can achieve performance comparable to PLMs (Pre-trained Language Models) without additional training, and PASS has a positive effect on the metaphor competence of LLMs.</abstract>
    <identifier type="citekey">gao-etal-2025-consistency</identifier>
    <location>
        <url>https://aclanthology.org/2025.coling-main.697/</url>
    </location>
    <part>
        <date>2025-01</date>
        <extent unit="page">
            <start>10460</start>
            <end>10471</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Consistency Rating of Semantic Transparency: an Evaluation Method for Metaphor Competence in Idiom Understanding Tasks
%A Gao, Hui
%A Zhang, Jing
%A Zhang, Peng
%A Yang, Chang
%Y Rambow, Owen
%Y Wanner, Leo
%Y Apidianaki, Marianna
%Y Al-Khalifa, Hend
%Y Eugenio, Barbara Di
%Y Schockaert, Steven
%S Proceedings of the 31st International Conference on Computational Linguistics
%D 2025
%8 January
%I Association for Computational Linguistics
%C Abu Dhabi, UAE
%F gao-etal-2025-consistency
%X Idioms condense complex semantics into fixed phrases, and their meaning is often not directly connected to the literal meaning of their constituent words, making idiom comprehension a test of metaphor competence. Metaphor, as a cognitive process in human beings, has not yet found an effective evaluation method to assess the metaphor competence of LLMs (Large Language Models). In this paper, we propose a method to evaluate the metaphor competence of LLMs for the idiom understanding task: the Consistency Rating of Semantic Transparency (CR-ST). This strategy assesses the difficulty of understanding idioms through two dimensions: overall semantic transparency and constituent semantic transparency, aiming to gauge LLMs’ mastery of metaphor competence. Subsequently, we introduce a prompt mechanism-Paraphrase Augmentation Strategy with Self-checking (PASS), based on human language logic, which guides the model to enhance its metaphor competence by explicitly generating idiom paraphrases. We conducted a baseline evaluation of seven LLMs on the CINLID and ChID datasets and analyzed the effectiveness of PASS on different subsets of semantic transparency. The experimental results demonstrate that LLMs can achieve performance comparable to PLMs (Pre-trained Language Models) without additional training, and PASS has a positive effect on the metaphor competence of LLMs.
%U https://aclanthology.org/2025.coling-main.697/
%P 10460-10471

Markdown (Informal)

[Consistency Rating of Semantic Transparency: an Evaluation Method for Metaphor Competence in Idiom Understanding Tasks](https://aclanthology.org/2025.coling-main.697/) (Gao et al., COLING 2025)

• Consistency Rating of Semantic Transparency: an Evaluation Method for Metaphor Competence in Idiom Understanding Tasks (Gao et al., COLING 2025)

ACL

• Hui Gao, Jing Zhang, Peng Zhang, and Chang Yang. 2025. Consistency Rating of Semantic Transparency: an Evaluation Method for Metaphor Competence in Idiom Understanding Tasks. In Proceedings of the 31st International Conference on Computational Linguistics, pages 10460–10471, Abu Dhabi, UAE. Association for Computational Linguistics.