<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="qian-etal-2024-large-language">
    <titleInfo>
        <title>Are Large Language Models State-of-the-art Quality Estimators for Machine Translation of User-generated Content?</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Shenbin</namePart>
        <namePart type="family">Qian</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Constantin</namePart>
        <namePart type="family">Orasan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Diptesh</namePart>
        <namePart type="family">Kanojia</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Félix</namePart>
        <namePart type="family">Do Carmo</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Eleventh Workshop on Asian Translation (WAT 2024)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Toshiaki</namePart>
            <namePart type="family">Nakazawa</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Isao</namePart>
            <namePart type="family">Goto</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Miami, Florida, USA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>This paper investigates whether large language models (LLMs) are state-of-the-art quality estimators for machine translation of user-generated content (UGC) that contains emotional expressions, without the use of reference translations. To achieve this, we employ an existing emotion-related dataset with human-annotated errors and calculate quality evaluation scores based on the Multi-dimensional Quality Metrics. We compare the accuracy of several LLMs with that of our fine-tuned baseline models, under in-context learning and parameter-efficient fine-tuning (PEFT) scenarios. We find that PEFT of LLMs leads to better performance in score prediction with human interpretable explanations than fine-tuned models. However, a manual analysis of LLM outputs reveals that they still have problems such as refusal to reply to a prompt and unstable output while evaluating machine translation of UGC.</abstract>
    <identifier type="citekey">qian-etal-2024-large-language</identifier>
    <location>
        <url>https://aclanthology.org/2024.wat-1.4</url>
    </location>
    <part>
        <date>2024-11</date>
        <extent unit="page">
            <start>45</start>
            <end>55</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Are Large Language Models State-of-the-art Quality Estimators for Machine Translation of User-generated Content?
%A Qian, Shenbin
%A Orasan, Constantin
%A Kanojia, Diptesh
%A Do Carmo, Félix
%Y Nakazawa, Toshiaki
%Y Goto, Isao
%S Proceedings of the Eleventh Workshop on Asian Translation (WAT 2024)
%D 2024
%8 November
%I Association for Computational Linguistics
%C Miami, Florida, USA
%F qian-etal-2024-large-language
%X This paper investigates whether large language models (LLMs) are state-of-the-art quality estimators for machine translation of user-generated content (UGC) that contains emotional expressions, without the use of reference translations. To achieve this, we employ an existing emotion-related dataset with human-annotated errors and calculate quality evaluation scores based on the Multi-dimensional Quality Metrics. We compare the accuracy of several LLMs with that of our fine-tuned baseline models, under in-context learning and parameter-efficient fine-tuning (PEFT) scenarios. We find that PEFT of LLMs leads to better performance in score prediction with human interpretable explanations than fine-tuned models. However, a manual analysis of LLM outputs reveals that they still have problems such as refusal to reply to a prompt and unstable output while evaluating machine translation of UGC.
%U https://aclanthology.org/2024.wat-1.4
%P 45-55

Markdown (Informal)

[Are Large Language Models State-of-the-art Quality Estimators for Machine Translation of User-generated Content?](https://aclanthology.org/2024.wat-1.4) (Qian et al., WAT 2024)

• Are Large Language Models State-of-the-art Quality Estimators for Machine Translation of User-generated Content? (Qian et al., WAT 2024)

ACL

• Shenbin Qian, Constantin Orasan, Diptesh Kanojia, and Félix Do Carmo. 2024. Are Large Language Models State-of-the-art Quality Estimators for Machine Translation of User-generated Content?. In Proceedings of the Eleventh Workshop on Asian Translation (WAT 2024), pages 45–55, Miami, Florida, USA. Association for Computational Linguistics.