Kamilia Rahmouni
Author directory2026
Cultural Understanding in Large Language Models: A Comparative Study with Human Judgments
Elsayed Issa | Kamilia Rahmouni
Proceedings of The Fourth Arabic Natural Language Processing Conference
Elsayed Issa | Kamilia Rahmouni
Proceedings of The Fourth Arabic Natural Language Processing Conference
This paper evaluates whether large language models (LLMs) can demonstrate cultural competence in Egyptian Arabic—not just factual knowledge about culture, but the ability to choose culturally appropriate idiomatic and figurative expressions in everyday social situations. 67 Egyptian participants completed a multiple-choice questionnaire of 90 everyday scenarios to provide human gold labels. Using the same scenarios, each pairing a culturally appropriate Egyptian idiom against a plausible distractor, we evaluated 11 open-source and 3 proprietary models under five prompts and two answer orderings. The results indicated that humans performed near ceiling (98.2%), while LLMs averaged 68.2%. The results also showed that a single accuracy figure conflates three separable behaviors: cultural competence, an answer-position heuristic, and abstention. These behaviors are distributed so unevenly that the model ranking reorders depending on which one is measured. One model’s apparent mid-tier accuracy is almost entirely a “pick the first option” heuristic (a 0.75 normal–reversed gap), and another falls from 51.8% to below chance once refusals to answer are scored as errors. We argue that cultural-competence evaluation should report this decomposition rather than a headline accuracy.