@inproceedings{zhang-etal-2026-hallomtbench,
title = "{H}allo{MTB}ench: A Large-Scale Human-Verified Benchmark for Diagnosing Hallucination Triggers in {LLM} Translation",
author = "Zhang, Shaowei and
Liu, Heng and
Wu, Xinwei and
Wang, Hao and
Liu, Yangyang and
Zhao, Xiaohu and
Peng, Jianxiang and
Shen, Tianhao and
Zhou, Jiang and
Zeng, Bo and
Dong, Tianyu and
Shi, Dan and
Xu, Linlong and
Wang, Longyue and
Xiong, Deyi and
Luo, Weihua",
editor = "Lee, Hung-Yi and
Dev, Sunipa and
Li, Jessy and
Elliott, Desmond",
booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
month = oct,
year = "2026",
address = "Budapest, Hungary",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.emnlp-main.1546/",
pages = "35261--35281",
ISBN = "979-8-89176-459-0",
abstract = "Large Language Models (LLMs) have advanced Machine Translation (MT), yet they remain vulnerable by hallucinations. Most existing benchmarks, originally designed for neural MT models, do not effectively elicit hallucinations from multilingual LLMs, resulting in a low density of samples for LLM translation hallucination analysis. To address this, we create HalloMTBench, a human-verified hallucination benchmark derived from tens of millions of translations, for studying signals that trigger LLM translation hallucinations across 11 English-to-X directions. We curate 6,908 authentic hallucination instances through a rigorous process: frontier LLMs first translate a large source corpus, and their outputs are then validated by both an LLM-judge ensemble and human experts. Based on our analysis of these hallucination instances, we introduce a taxonomy for translation hallucination that distinguishes Instruction Detachment from Source Detachment. Our evaluation of 21 model variants on HalloMTBench reveals several observable vulnerability patterns, including model Characteristics, source length, language proximity, hallucination reproducibility, and model-dependent effects of reasoning-style decoding. HalloMTBench offers a forward-looking testbed for diagnosing LLM translation hallucination. All resources will be released."
}<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="zhang-etal-2026-hallomtbench">
<titleInfo>
<title>HalloMTBench: A Large-Scale Human-Verified Benchmark for Diagnosing Hallucination Triggers in LLM Translation</title>
</titleInfo>
<name type="personal">
<namePart type="given">Shaowei</namePart>
<namePart type="family">Zhang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Heng</namePart>
<namePart type="family">Liu</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Xinwei</namePart>
<namePart type="family">Wu</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Hao</namePart>
<namePart type="family">Wang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Yangyang</namePart>
<namePart type="family">Liu</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Xiaohu</namePart>
<namePart type="family">Zhao</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Jianxiang</namePart>
<namePart type="family">Peng</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Tianhao</namePart>
<namePart type="family">Shen</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Jiang</namePart>
<namePart type="family">Zhou</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Bo</namePart>
<namePart type="family">Zeng</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Tianyu</namePart>
<namePart type="family">Dong</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Dan</namePart>
<namePart type="family">Shi</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Linlong</namePart>
<namePart type="family">Xu</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Longyue</namePart>
<namePart type="family">Wang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Deyi</namePart>
<namePart type="family">Xiong</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Weihua</namePart>
<namePart type="family">Luo</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<originInfo>
<dateIssued>2026-10</dateIssued>
</originInfo>
<typeOfResource>text</typeOfResource>
<relatedItem type="host">
<titleInfo>
<title>Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing</title>
</titleInfo>
<name type="personal">
<namePart type="given">Hung-Yi</namePart>
<namePart type="family">Lee</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Sunipa</namePart>
<namePart type="family">Dev</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Jessy</namePart>
<namePart type="family">Li</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Desmond</namePart>
<namePart type="family">Elliott</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<originInfo>
<publisher>Association for Computational Linguistics</publisher>
<place>
<placeTerm type="text">Budapest, Hungary</placeTerm>
</place>
</originInfo>
<genre authority="marcgt">conference publication</genre>
<identifier type="isbn">979-8-89176-459-0</identifier>
</relatedItem>
<abstract>Large Language Models (LLMs) have advanced Machine Translation (MT), yet they remain vulnerable by hallucinations. Most existing benchmarks, originally designed for neural MT models, do not effectively elicit hallucinations from multilingual LLMs, resulting in a low density of samples for LLM translation hallucination analysis. To address this, we create HalloMTBench, a human-verified hallucination benchmark derived from tens of millions of translations, for studying signals that trigger LLM translation hallucinations across 11 English-to-X directions. We curate 6,908 authentic hallucination instances through a rigorous process: frontier LLMs first translate a large source corpus, and their outputs are then validated by both an LLM-judge ensemble and human experts. Based on our analysis of these hallucination instances, we introduce a taxonomy for translation hallucination that distinguishes Instruction Detachment from Source Detachment. Our evaluation of 21 model variants on HalloMTBench reveals several observable vulnerability patterns, including model Characteristics, source length, language proximity, hallucination reproducibility, and model-dependent effects of reasoning-style decoding. HalloMTBench offers a forward-looking testbed for diagnosing LLM translation hallucination. All resources will be released.</abstract>
<identifier type="citekey">zhang-etal-2026-hallomtbench</identifier>
<location>
<url>https://aclanthology.org/2026.emnlp-main.1546/</url>
</location>
<part>
<date>2026-10</date>
<extent unit="page">
<start>35261</start>
<end>35281</end>
</extent>
</part>
</mods>
</modsCollection>
%0 Conference Proceedings
%T HalloMTBench: A Large-Scale Human-Verified Benchmark for Diagnosing Hallucination Triggers in LLM Translation
%A Zhang, Shaowei
%A Liu, Heng
%A Wu, Xinwei
%A Wang, Hao
%A Liu, Yangyang
%A Zhao, Xiaohu
%A Peng, Jianxiang
%A Shen, Tianhao
%A Zhou, Jiang
%A Zeng, Bo
%A Dong, Tianyu
%A Shi, Dan
%A Xu, Linlong
%A Wang, Longyue
%A Xiong, Deyi
%A Luo, Weihua
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-459-0
%F zhang-etal-2026-hallomtbench
%X Large Language Models (LLMs) have advanced Machine Translation (MT), yet they remain vulnerable by hallucinations. Most existing benchmarks, originally designed for neural MT models, do not effectively elicit hallucinations from multilingual LLMs, resulting in a low density of samples for LLM translation hallucination analysis. To address this, we create HalloMTBench, a human-verified hallucination benchmark derived from tens of millions of translations, for studying signals that trigger LLM translation hallucinations across 11 English-to-X directions. We curate 6,908 authentic hallucination instances through a rigorous process: frontier LLMs first translate a large source corpus, and their outputs are then validated by both an LLM-judge ensemble and human experts. Based on our analysis of these hallucination instances, we introduce a taxonomy for translation hallucination that distinguishes Instruction Detachment from Source Detachment. Our evaluation of 21 model variants on HalloMTBench reveals several observable vulnerability patterns, including model Characteristics, source length, language proximity, hallucination reproducibility, and model-dependent effects of reasoning-style decoding. HalloMTBench offers a forward-looking testbed for diagnosing LLM translation hallucination. All resources will be released.
%U https://aclanthology.org/2026.emnlp-main.1546/
%P 35261-35281
Markdown (Informal)
[HalloMTBench: A Large-Scale Human-Verified Benchmark for Diagnosing Hallucination Triggers in LLM Translation](https://aclanthology.org/2026.emnlp-main.1546/) (Zhang et al., EMNLP 2026)
ACL
- Shaowei Zhang, Heng Liu, Xinwei Wu, Hao Wang, Yangyang Liu, Xiaohu Zhao, Jianxiang Peng, Tianhao Shen, Jiang Zhou, Bo Zeng, Tianyu Dong, Dan Shi, Linlong Xu, Longyue Wang, Deyi Xiong, and Weihua Luo. 2026. HalloMTBench: A Large-Scale Human-Verified Benchmark for Diagnosing Hallucination Triggers in LLM Translation. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 35261–35281, Budapest, Hungary. Association for Computational Linguistics.