From Synthetic to Localized: Benchmarking Multilingual Intent Classification in Logistics Customer Service

Haoyu He, Jinyu Zhuang, Haoran Chu, Shuhang yu, Hao Wang, Han Kunpeng


Abstract
Multilingual intent classification is central to logistics customer-service systems, where models must handle noisy user queries across languages and hierarchical labels. Yet many multilingual benchmarks rely on machine-translated text, which is cleaner than real customer requests and can overestimate robustness. We present a public benchmark for hierarchical multilingual intent classification from real logistics logs: approximately 30K de-identified, single-utterance queries curated from 600K records through filtering, LLM-assisted quality control, human privacy review, and sampled label auditing. The dataset uses a two-level taxonomy with 13 parent and 17 leaf intents; English, Spanish, and Arabic are seen languages, while Indonesian, Chinese, and Thai support zero-shot evaluation. To measure the gap between machine-translated and professionally localized evaluation, we provide paired test sets derived from the same source items. We benchmark multilingual encoders, embedding models, and small language models under flat and hierarchical protocols, and find that machine translation tests yield higher accuracy in most settings. [Prompts and sample data are available at <https://github.com/JTExpress-MICCS/MICCS>. A preprint version of this work is available at <https://arxiv.org/abs/2603.23172>.]
Anthology ID:
2026.emnlp-industry.121
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Pepa Atanasova, Gholamreza Haffari, Minghan Wang
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
2078–2092
Language:
URL:
https://aclanthology.org/2026.emnlp-industry.121/
DOI:
Bibkey:
Cite (ACL):
Haoyu He, Jinyu Zhuang, Haoran Chu, Shuhang yu, Hao Wang, and Han Kunpeng. 2026. From Synthetic to Localized: Benchmarking Multilingual Intent Classification in Logistics Customer Service. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 2078–2092, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
From Synthetic to Localized: Benchmarking Multilingual Intent Classification in Logistics Customer Service (He et al., EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-industry.121.pdf