Hernan Contreras
2026
Leveraging Wikidata for Geographically Informed Sociocultural Bias Dataset Creation: Application to Latin America
Yannis Karmim | Renato Pino | Hernan Contreras | Hernan Lira | Sebastian Cifuentes | Simon Escoffier | Luis Martí | Djamé Seddah | Valentin Barriere
Proceedings of the First Workshop on Multilingual Multicultural Evaluation
Yannis Karmim | Renato Pino | Hernan Contreras | Hernan Lira | Sebastian Cifuentes | Simon Escoffier | Luis Martí | Djamé Seddah | Valentin Barriere
Proceedings of the First Workshop on Multilingual Multicultural Evaluation
Large Language Models (LLMs) exhibit inequalities with respect to various cultural contexts. Most prominent open-weights models are trained on Global North data and show prejudicial behavior towards other cultures. Moreover, there is a notable lack of resources to detect biases in non-English languages, especially from Latin America (Latam), a continent containing various cultures, even though they share a common cultural ground.We propose to leverage the content of Wikipedia, the structure of the Wikidata knowledge graph, and expert knowledge from social science in order to create a dataset of Questions/Answers (Q/As) pairs, based on the different popular and social cultures of various Latin American countries. We create a database of around 23k questions and associated answers extracted from 23k Wikipedia articles, and transformed into a multiple-choice questions (MCQ) in Spanish and Portuguese, in turn translated to English. We use this MCQ to quantify the degree of knowledge of various LLMs and find out extit(i) a discrepancy in performances between the Latam countries, ones being easier than others for the majority of the models, extit(ii) that the models perform better in their original language, extit(iii) that Iberian Spanish culture is better known than Latam one. Our code, our results for reproducing the results, and all datasets by region will be available.