Tagged Span Annotation for Detecting Translation Errors in Reasoning LLMs

Taemin Yeom; Yonghyun Ryu; Yoonjung Choi; Jinyeong Bak

Tagged Span Annotation for Detecting Translation Errors in Reasoning LLMs

Taemin Yeom, Yonghyun Ryu, Yoonjung Choi, Jinyeong Bak

Correct Metadata for

Important: The Anthology treat PDFs as authoritative. Please use this form only to correct data that is out of line with the PDF. See our corrections guidelines if you need to change the PDF.

Title Adjust the title. Retain tags such as <fixed-case>.

Authors Adjust author names and order to match the PDF.

Abstract Correct abstract if needed. Retain XML formatting tags such as <tex-math>.

Verification against PDF Ensure that the new title/authors match the snapshot below. (If there is no snapshot or it is too small, consult the PDF.)

Authors concatenated from the text boxes above:

ALL author names match the snapshot above—including middle initials, hyphens, and accents.

Abstract

We present the AIP team’s submission to the WMT 2025 Unified MT Evaluation SharedTask, focusing on the span-level error detection subtask. Our system emphasizes response format design to better harness the capabilities of OpenAI’s o3, the state-of-the-art reasoning LLM. To this end, we introduce Tagged SpanAnnotation (TSA), an annotation scheme designed to more accurately extract span-level information from the LLM. On our refined version of WMT24 ESA dataset, our reference-free method achieves an F1 score of approximately 27 for character-level label prediction, outperforming the reference-based XCOMET-XXL at approximately 17.

Anthology ID:: 2025.wmt-1.62
Volume:: Proceedings of the Tenth Conference on Machine Translation
Month:: November
Year:: 2025
Address:: Suzhou, China
Editors:: Barry Haddow, Tom Kocmi, Philipp Koehn, Christof Monz
Venue:: WMT
SIG:
Publisher:: Association for Computational Linguistics
Note:
Pages:: 878–886
Language:
URL:: https://aclanthology.org/2025.wmt-1.62/
DOI:
Bibkey:
Cite (ACL):: Taemin Yeom, Yonghyun Ryu, Yoonjung Choi, and Jinyeong Bak. 2025. Tagged Span Annotation for Detecting Translation Errors in Reasoning LLMs. In Proceedings of the Tenth Conference on Machine Translation, pages 878–886, Suzhou, China. Association for Computational Linguistics.
Cite (Informal):: Tagged Span Annotation for Detecting Translation Errors in Reasoning LLMs (Yeom et al., WMT 2025)
Copy Citation:
PDF:: https://aclanthology.org/2025.wmt-1.62.pdf

PDF Cite Search Fix data

Export citation

BibTeX
MODS XML
Endnote
Preformatted

@inproceedings{yeom-etal-2025-tagged,
    title = "Tagged Span Annotation for Detecting Translation Errors in Reasoning {LLM}s",
    author = "Yeom, Taemin  and
      Ryu, Yonghyun  and
      Choi, Yoonjung  and
      Bak, Jinyeong",
    editor = "Haddow, Barry  and
      Kocmi, Tom  and
      Koehn, Philipp  and
      Monz, Christof",
    booktitle = "Proceedings of the Tenth Conference on Machine Translation",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.wmt-1.62/",
    pages = "878--886",
    ISBN = "979-8-89176-341-8",
    abstract = "We present the AIP team{'}s submission to the WMT 2025 Unified MT Evaluation SharedTask, focusing on the span-level error detection subtask. Our system emphasizes response format design to better harness the capabilities of OpenAI{'}s o3, the state-of-the-art reasoning LLM. To this end, we introduce Tagged SpanAnnotation (TSA), an annotation scheme designed to more accurately extract span-level information from the LLM. On our refined version of WMT24 ESA dataset, our reference-free method achieves an F1 score of approximately 27 for character-level label prediction, outperforming the reference-based XCOMET-XXL at approximately 17."
}

Download as File

<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="yeom-etal-2025-tagged">
    <titleInfo>
        <title>Tagged Span Annotation for Detecting Translation Errors in Reasoning LLMs</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Taemin</namePart>
        <namePart type="family">Yeom</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yonghyun</namePart>
        <namePart type="family">Ryu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yoonjung</namePart>
        <namePart type="family">Choi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jinyeong</namePart>
        <namePart type="family">Bak</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Tenth Conference on Machine Translation</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Barry</namePart>
            <namePart type="family">Haddow</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Tom</namePart>
            <namePart type="family">Kocmi</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Philipp</namePart>
            <namePart type="family">Koehn</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Christof</namePart>
            <namePart type="family">Monz</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Suzhou, China</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-341-8</identifier>
    </relatedItem>
    <abstract>We present the AIP team’s submission to the WMT 2025 Unified MT Evaluation SharedTask, focusing on the span-level error detection subtask. Our system emphasizes response format design to better harness the capabilities of OpenAI’s o3, the state-of-the-art reasoning LLM. To this end, we introduce Tagged SpanAnnotation (TSA), an annotation scheme designed to more accurately extract span-level information from the LLM. On our refined version of WMT24 ESA dataset, our reference-free method achieves an F1 score of approximately 27 for character-level label prediction, outperforming the reference-based XCOMET-XXL at approximately 17.</abstract>
    <identifier type="citekey">yeom-etal-2025-tagged</identifier>
    <location>
        <url>https://aclanthology.org/2025.wmt-1.62/</url>
    </location>
    <part>
        <date>2025-11</date>
        <extent unit="page">
            <start>878</start>
            <end>886</end>
        </extent>
    </part>
</mods>
</modsCollection>

Download as File

%0 Conference Proceedings
%T Tagged Span Annotation for Detecting Translation Errors in Reasoning LLMs
%A Yeom, Taemin
%A Ryu, Yonghyun
%A Choi, Yoonjung
%A Bak, Jinyeong
%Y Haddow, Barry
%Y Kocmi, Tom
%Y Koehn, Philipp
%Y Monz, Christof
%S Proceedings of the Tenth Conference on Machine Translation
%D 2025
%8 November
%I Association for Computational Linguistics
%C Suzhou, China
%@ 979-8-89176-341-8
%F yeom-etal-2025-tagged
%X We present the AIP team’s submission to the WMT 2025 Unified MT Evaluation SharedTask, focusing on the span-level error detection subtask. Our system emphasizes response format design to better harness the capabilities of OpenAI’s o3, the state-of-the-art reasoning LLM. To this end, we introduce Tagged SpanAnnotation (TSA), an annotation scheme designed to more accurately extract span-level information from the LLM. On our refined version of WMT24 ESA dataset, our reference-free method achieves an F1 score of approximately 27 for character-level label prediction, outperforming the reference-based XCOMET-XXL at approximately 17.
%U https://aclanthology.org/2025.wmt-1.62/
%P 878-886

Download as File

Markdown (Informal)

[Tagged Span Annotation for Detecting Translation Errors in Reasoning LLMs](https://aclanthology.org/2025.wmt-1.62/) (Yeom et al., WMT 2025)

Tagged Span Annotation for Detecting Translation Errors in Reasoning LLMs (Yeom et al., WMT 2025)

ACL

Taemin Yeom, Yonghyun Ryu, Yoonjung Choi, and Jinyeong Bak. 2025. Tagged Span Annotation for Detecting Translation Errors in Reasoning LLMs. In Proceedings of the Tenth Conference on Machine Translation, pages 878–886, Suzhou, China. Association for Computational Linguistics.