Juan J. Calderón G.
Author directory2026
INCLUDE-2.0: Disentangling Knowledge Gaps in Multilingual LLM Evaluation
Angelika Romanou | María Grandury | Clara Meister | Negar Foroutan | Jabez Magomere | Anna Sotnikova | Hitesh Laxmichand Patel | Akshay Kumar Dubey | Alfonso Amayuelas | Amit Agarwal | Arion Das | Carlos Arriaga | Cheng Yu | Dalia Ali | David Quispe-Castillo | Fenal Ashokbhai Ilasariya | Francisco-Javier Rodrigo-Ginés | Gonzalo Martínez | Gouthami Vadithya | Guido Ivetta | Javier Conde | Jorge Vallego | Juan J. Calderón G. | Karan Gupta | Marcos J Gomez | Marvin M. Agüero-Torales | Mithil Bangera | Naira Paola Arnez Jordan | Oscar M. Cumbicus-Pineda | Priyaranjan Pattnayak | Reewos Erwin Talla Chumpitaz | Sahil Aggarwal | Sahil Raj | Shruti Jalan | Srikant Panda | Susana Zhou | Yeshil Bangera | Orestis Papakyriakopoulos | Scott A. Hale | Shamsuddeen Hassan Muhammad | Antoine Bosselut
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Angelika Romanou | María Grandury | Clara Meister | Negar Foroutan | Jabez Magomere | Anna Sotnikova | Hitesh Laxmichand Patel | Akshay Kumar Dubey | Alfonso Amayuelas | Amit Agarwal | Arion Das | Carlos Arriaga | Cheng Yu | Dalia Ali | David Quispe-Castillo | Fenal Ashokbhai Ilasariya | Francisco-Javier Rodrigo-Ginés | Gonzalo Martínez | Gouthami Vadithya | Guido Ivetta | Javier Conde | Jorge Vallego | Juan J. Calderón G. | Karan Gupta | Marcos J Gomez | Marvin M. Agüero-Torales | Mithil Bangera | Naira Paola Arnez Jordan | Oscar M. Cumbicus-Pineda | Priyaranjan Pattnayak | Reewos Erwin Talla Chumpitaz | Sahil Aggarwal | Sahil Raj | Shruti Jalan | Srikant Panda | Susana Zhou | Yeshil Bangera | Orestis Papakyriakopoulos | Scott A. Hale | Shamsuddeen Hassan Muhammad | Antoine Bosselut
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Current multilingual benchmarks cover only a fraction of the languages represented in modern pretraining corpora and often conflate language proficiency with the ability to answer questions grounded in specific regions and communities. We introduce INCLUDE-2.0, a large-scale benchmark for region-aware multilingual factual evaluation, which covers 91 languages from 86 countries (127 total language–country pairs), comprising over 220K curated samples drawn from native regional sources. Unlike existing benchmarks that often treat language as a proxy for region, IINCLUDE-2.0 distinguishes between questions whose answers are independent of context, vary by country, concern widely documented facts about a place, or depend on knowledge primarily expressed through local linguistic and cultural sources. Each item is annotated with its regional knowledge type and domain, and paired with an English translation, enabling controlled analysis of the relationship between language proficiency and possession of regional knowledge.
Evaluating frontier and open-weight models on IINCLUDE-2.0, we find that aggregate multilingual accuracy masks substantial variation across regional knowledge types, languages, and regions. We find that frontier models perform worse on locally embedded knowledge than on widely documented facts about regions, with an average drop of ~14.5 points. We also show that for open-weight models, scale improves accuracy but does not reliably make performance invariant to whether a question is asked in English or its original language. INCLUDE-2.0 is available at https://huggingface.co/datasets/epfl-nlp/include-2.0
Evaluating frontier and open-weight models on IINCLUDE-2.0, we find that aggregate multilingual accuracy masks substantial variation across regional knowledge types, languages, and regions. We find that frontier models perform worse on locally embedded knowledge than on widely documented facts about regions, with an average drop of ~14.5 points. We also show that for open-weight models, scale improves accuracy but does not reliably make performance invariant to whether a question is asked in English or its original language. INCLUDE-2.0 is available at https://huggingface.co/datasets/epfl-nlp/include-2.0
Search
Fix author
Co-authors
- Amit Agarwal 1
- Sahil Aggarwal 1
- Marvin Agüero-Torales 1
- Dalia Ali 1
- Alfonso Amayuelas 1
- Carlos Arriaga 1
- Mithil Bangera 1
- Yeshil Bangera 1
- Antoine Bosselut 1
- Reewos Erwin Talla Chumpitaz 1
- Javier Conde 1
- Oscar M. Cumbicus-Pineda 1
- Arion Das 1
- Akshay Kumar Dubey 1
- Negar Foroutan 1
- Marcos J Gomez 1
- María Grandury 1
- Karan Gupta 1
- Scott A. Hale 1
- Fenal Ashokbhai Ilasariya 1
- Guido Ivetta 1
- Shruti Jalan 1
- Naira Paola Arnez Jordan 1
- Jabez Magomere 1
- Gonzalo Martínez 1
- Clara Meister 1
- Shamsuddeen Hassan Muhammad 1
- Srikant Panda 1
- Orestis Papakyriakopoulos 1
- Hitesh Laxmichand Patel 1
- Priyaranjan Pattnayak 1
- David Quispe-Castillo 1
- Sahil Raj 1
- Francisco-Javier Rodrigo-Ginés 1
- Angelika Romanou 1
- Anna Sotnikova 1
- Gouthami Vadithya 1
- Jorge Vallego 1
- Cheng Yu 1
- Susana Zhou 1