Thong T. Doan
Author directory2026
RegioZH: Unveiling Comprehension Disparities across Regional Chinese Varieties in LLMs
Weihua Zheng | Wu Kui | Thong T. Doan | Lin Jingxia | AiTi Aw | Nancy F. Chen | Roy Ka-Wei Lee
Findings of the Association for Computational Linguistics: EMNLP 2026
Weihua Zheng | Wu Kui | Thong T. Doan | Lin Jingxia | AiTi Aw | Nancy F. Chen | Roy Ka-Wei Lee
Findings of the Association for Computational Linguistics: EMNLP 2026
Current evaluations often treat Chinese varieties as monolithic, obscuring failures for regional users. We introduce RegioZH, a 2,607-item multiple-choice benchmark for regional lexical reasoning across Mainland China, Taiwan, Hong Kong, Singapore, and Malaysia. Built by region-specific annotators and expert-validated, it targets localized idioms, slang, code-mixing, politeness conventions, and sociocultural expressions. Zero-shot results on eleven LLMs show large disparities, including a 29.4-point average accuracy gap between Taiwan and Singapore. Persona and structured chain-of-thought prompts for five models yield selective gains, suggesting that some errors stem from retrieval or framing rather than only missing knowledge. Representation, attention, calibration, and surface-feature analyses show a dissociation between marker attention and semantic grounding, overconfident forced-choice errors, and localized lexical density as a consistent difficulty predictor. Qualitative analysis finds terminology misunderstandings, sociocultural inference gaps, and cross-regional semantic divergence. RegioZH demonstrates that apparent Chinese proficiency can mask uneven understanding across regional varieties, motivating intra-language robustness evaluation alongside multilingual generalization.
CultureConverse: A Multilingual Multi-turn Simulation Harness for Culturally Grounded Assistance in East and Southeast Asia
Bryan Chen Zhengyu Tan | Weihua Zheng | Thong T. Doan | Bich Ngoc Doan | Jia Wang Peh | Xiaoyuan Yi | Jing Yao | Xing Xie | Nancy F. Chen | Zhengyuan Liu | JinYeong Bak | Wafi Shamdi | Soo Kai Chie | Liew Yu Siong | Aina Azyyati Binti Mohamad Rezal | Lew Yan Yan Vanessa | Huadan Wu | Dylan Raharja | Nadya Yuki Wangsajaya | Akane Fukushige | Kazushi Kato | Koji Inoue | Tatsuya Kawahara | Jaehyung Seo | Dongjun Kim | Seungyoon Lee | Zi Haur Pang | Rui Yang Tan | Charibeth Ko Cheng | Maria Regina Justina Estuar | Jann Railey Montalan | Pham Minh Duc | Roy Ka-Wei Lee
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Bryan Chen Zhengyu Tan | Weihua Zheng | Thong T. Doan | Bich Ngoc Doan | Jia Wang Peh | Xiaoyuan Yi | Jing Yao | Xing Xie | Nancy F. Chen | Zhengyuan Liu | JinYeong Bak | Wafi Shamdi | Soo Kai Chie | Liew Yu Siong | Aina Azyyati Binti Mohamad Rezal | Lew Yan Yan Vanessa | Huadan Wu | Dylan Raharja | Nadya Yuki Wangsajaya | Akane Fukushige | Kazushi Kato | Koji Inoue | Tatsuya Kawahara | Jaehyung Seo | Dongjun Kim | Seungyoon Lee | Zi Haur Pang | Rui Yang Tan | Charibeth Ko Cheng | Maria Regina Justina Estuar | Jann Railey Montalan | Pham Minh Duc | Roy Ka-Wei Lee
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Current cultural evaluations for large language models (LLMs) often reduce culture to single-turn factual recall via MCQs, failing to capture a common use case: users seeking practical help over multiple turns in culturally grounded scenarios. We introduce CultureConverse, a scalable, multilingual simulation and evaluation harness for culturally grounded assistant dialogue that covers 10 East and Southeast Asian regions, 58 subgroup identities, and 7 domains. Each simulated and evaluated episode produces a scored interaction where the assistant assists the user and infers cultural constraints from partial information. The resulting CultureConverse-DS dataset contains 14,610 benchmark (evaluation) episodes and 274,295 oracle-guided (gold-mode) dialogues. In our benchmark evaluation of 18 models, GPT-5 mini achieves the highest assistance quality. Human annotation experiments suggest that our evaluation framework is a sufficient proxy for human judgment. Performance gains from fine-tuning on 27,860 high-quality CultureConverse-DS samples improve in-domain assistance and transfer out-of-domain to cultural MCQ and safety classification benchmarks. We release the harness, both splits, and judge prompts to support interactive evaluation of cultural competency.
Search
Fix author
Co-authors
- Nancy Chen 2
- Roy Ka-Wei Lee 2
- Zheng Weihua 2
- Aiti Aw 1
- JinYeong Bak 1
- Charibeth Ko Cheng 1
- Soo Kai Chie 1
- Bich Ngoc Doan 1
- Pham Minh Duc 1
- Maria Regina Justina Estuar 1
- Akane Fukushige 1
- Koji Inoue 1
- Lin Jingxia 1
- Kazushi Kato 1
- Tatsuya Kawahara 1
- Dongjun Kim 1
- Wu Kui 1
- Seungyoon Lee 1
- Zhengyuan Liu 1
- Jann Railey Montalan 1
- Zi Haur Pang 1
- Jia Wang Peh 1
- Dylan Raharja 1
- Aina Azyyati Binti Mohamad Rezal 1
- Jaehyung Seo 1
- Wafi Shamdi 1
- Liew Yu Siong 1
- Bryan Chen Zhengyu Tan 1
- Rui Yang Tan 1
- Lew Yan Yan Vanessa 1
- Nadya Yuki Wangsajaya 1
- Huadan Wu 1
- Xing Xie 1
- Jing Yao 1
- Xiaoyuan Yi 1