Sarakmatak Sophy
Author directory2026
CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks
Peiqin Lin | Chenyang Lyu | Wenjiang Luo | Haotian Ye | Md Mehrab Hossain | Chunlan Ma | Shaoxiong Ji | Younes Samih | Bo Zeng | Fan Jiang | Yuanbin Cao | Dilda Duisenbek | Adrian Neo Sau Xun | Pozdniakova Daria | Misevich Liubou | Marinković Nevena | Ngoc Gia Linh Nguyen | Thi Khanh Linh DO | Sarakmatak Sophy | Baotian Hu | Guanhua Chen | Gongbo Tang | Alham Fikri Aji | Longyue Wang | Weihua Luo
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Peiqin Lin | Chenyang Lyu | Wenjiang Luo | Haotian Ye | Md Mehrab Hossain | Chunlan Ma | Shaoxiong Ji | Younes Samih | Bo Zeng | Fan Jiang | Yuanbin Cao | Dilda Duisenbek | Adrian Neo Sau Xun | Pozdniakova Daria | Misevich Liubou | Marinković Nevena | Ngoc Gia Linh Nguyen | Thi Khanh Linh DO | Sarakmatak Sophy | Baotian Hu | Guanhua Chen | Gongbo Tang | Alham Fikri Aji | Longyue Wang | Weihua Luo
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Large language models (LLMs) are now deployed worldwide, inspiring a surge of benchmarks that measure their multilingual and multicultural abilities. However, these benchmarks prioritize generic language understanding or superficial cultural trivia, leaving the evaluation of grounded tasks—where models must reason within real-world, context-rich scenarios—largely unaddressed. To fill this gap, we present CulturALL, a comprehensive and challenging benchmark to assess LLMs’ multilingual and multicultural competence on grounded tasks. CulturALL is built via a human–AI collaborative framework: expert annotators ensure appropriate difficulty and factual accuracy, while LLMs lighten the manual workload. By incorporating diverse sources, CulturALL ensures comprehensive scenario coverage. Each item is carefully designed to present a high level of difficulty, making CulturALL challenging. CulturALL contains 2,610 samples in 14 languages of 51 regions, distributed across 16 topics to capture the full breadth of grounded tasks. The experiments show that the best LLM achieve 44.48% accuracy on CulturALL, underscoring substantial room for improvement.