<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="vandan-etal-2023-towards">
    <titleInfo>
        <title>Towards Large Language Model driven Reference-less Translation Evaluation for English and Indian Language</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Mujadia</namePart>
        <namePart type="family">Vandan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Mishra</namePart>
        <namePart type="family">Pruthwik</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ahsan</namePart>
        <namePart type="family">Arafat</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">M</namePart>
        <namePart type="given">Sharma</namePart>
        <namePart type="family">Dipti</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 20th International Conference on Natural Language Processing (ICON)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">D</namePart>
            <namePart type="given">Pawar</namePart>
            <namePart type="family">Jyoti</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Lalitha</namePart>
            <namePart type="given">Devi</namePart>
            <namePart type="family">Sobha</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>NLP Association of India (NLPAI)</publisher>
            <place>
                <placeTerm type="text">Goa University, Goa, India</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>With the primary focus on evaluating the effectiveness of large language models for automatic reference-less translation assessment, this work presents our experiments on mimicking human direct assessment to evaluate the quality of translations in English and Indian languages. We constructed a translation evaluation task where we performed zero-shot learning, in-context example-driven learning, and fine-tuning of large language models to provide a score out of 100, where 100 represents a perfect translation and 1 represents a poor translation. We compared the performance of our trained systems with existing methods such as COMET, BERT-Scorer, and LABSE, and found that the LLM-based evaluator (LLaMA2-13B) achieves a comparable or higher overall correlation with human judgments for the considered Indian language pairs (Refer figure 1).</abstract>
    <identifier type="citekey">vandan-etal-2023-towards</identifier>
    <location>
        <url>https://aclanthology.org/2023.icon-1.28</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>357</start>
            <end>369</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Towards Large Language Model driven Reference-less Translation Evaluation for English and Indian Language
%A Vandan, Mujadia
%A Pruthwik, Mishra
%A Arafat, Ahsan
%A Dipti, M. Sharma
%Y Jyoti, D. Pawar
%Y Sobha, Lalitha Devi
%S Proceedings of the 20th International Conference on Natural Language Processing (ICON)
%D 2023
%8 December
%I NLP Association of India (NLPAI)
%C Goa University, Goa, India
%F vandan-etal-2023-towards
%X With the primary focus on evaluating the effectiveness of large language models for automatic reference-less translation assessment, this work presents our experiments on mimicking human direct assessment to evaluate the quality of translations in English and Indian languages. We constructed a translation evaluation task where we performed zero-shot learning, in-context example-driven learning, and fine-tuning of large language models to provide a score out of 100, where 100 represents a perfect translation and 1 represents a poor translation. We compared the performance of our trained systems with existing methods such as COMET, BERT-Scorer, and LABSE, and found that the LLM-based evaluator (LLaMA2-13B) achieves a comparable or higher overall correlation with human judgments for the considered Indian language pairs (Refer figure 1).
%U https://aclanthology.org/2023.icon-1.28
%P 357-369

Markdown (Informal)

[Towards Large Language Model driven Reference-less Translation Evaluation for English and Indian Language](https://aclanthology.org/2023.icon-1.28) (Vandan et al., ICON 2023)

• Towards Large Language Model driven Reference-less Translation Evaluation for English and Indian Language (Vandan et al., ICON 2023)

ACL

• Mujadia Vandan, Mishra Pruthwik, Ahsan Arafat, and M. Sharma Dipti. 2023. Towards Large Language Model driven Reference-less Translation Evaluation for English and Indian Language. In Proceedings of the 20th International Conference on Natural Language Processing (ICON), pages 357–369, Goa University, Goa, India. NLP Association of India (NLPAI).