Susu Box or Piggy Bank: Assessing Cultural Commonsense Knowledge between Ghana and the US

Christabel Acquaye, Haozhe An, Rachel Rudinger


Abstract
Recent work has highlighted the culturally-contingent nature of commonsense knowledge. We introduce AMAMMERε, a test set of 525 multiple-choice questions designed to evaluate the commonsense knowledge of English LLMs, relative to the cultural contexts of Ghana and the United States. To create AMAMMERε, we select a set of multiple-choice questions (MCQs) from existing commonsense datasets and rewrite them in a multi-stage process involving surveys of Ghanaian and U.S. participants. In three rounds of surveys, participants from both pools are solicited to (1) write correct and incorrect answer choices, (2) rate individual answer choices on a 5-point Likert scale, and (3) select the best answer choice from the newly-constructed MCQ items, in a final validation step. By engaging participants at multiple stages, our procedure ensures that participant perspectives are incorporated both in the creation and validation of test items, resulting in high levels of agreement within each pool. We evaluate several off-the-shelf English LLMs on AMAMMERε. Uniformly, models prefer answers choices that align with the preferences of U.S. annotators over Ghanaian annotators. Additionally, when test items specify a cultural context (Ghana or the U.S.), models exhibit some ability to adapt, but performance is consistently better in U.S. contexts than Ghanaian. As large resources are devoted to the advancement of English LLMs, our findings underscore the need for culturally adaptable models and evaluations to meet the needs of diverse English-speaking populations around the world.
Anthology ID:
2024.emnlp-main.532
Volume:
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2024
Address:
Miami, Florida, USA
Editors:
Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
9483–9502
Language:
URL:
https://aclanthology.org/2024.emnlp-main.532
DOI:
Bibkey:
Cite (ACL):
Christabel Acquaye, Haozhe An, and Rachel Rudinger. 2024. Susu Box or Piggy Bank: Assessing Cultural Commonsense Knowledge between Ghana and the US. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9483–9502, Miami, Florida, USA. Association for Computational Linguistics.
Cite (Informal):
Susu Box or Piggy Bank: Assessing Cultural Commonsense Knowledge between Ghana and the US (Acquaye et al., EMNLP 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.emnlp-main.532.pdf
Data:
 2024.emnlp-main.532.data.zip