Andrea Schimmenti
2026
A Wikidata-Based Framework to Measure Cross-Lingual Bias in Multilingual Large Language Models
Mouloud Iferroudjene | Lisa Poggel | Andrea Schimmenti | Duo Yang | Kanchan Shivashankar | Jan-Christoph Kalo | Marta Boscariol
Proceedings of the Knowledge Graphs and Large Language Models Workshop (KG-LLM) @ LREC26
Mouloud Iferroudjene | Lisa Poggel | Andrea Schimmenti | Duo Yang | Kanchan Shivashankar | Jan-Christoph Kalo | Marta Boscariol
Proceedings of the Knowledge Graphs and Large Language Models Workshop (KG-LLM) @ LREC26
Multilingual large language models (LLMs) are increasingly used for factual question answering, yet their accuracy varies across languages in ways that are difficult to interpret. A central challenge is that many multilingual probing benchmarks conflate multiple factors: the language used to ask the question, the cultural-linguistic context of the entities being queried, and the popularity skew of entities. In our paper, we disentangle these factors by asking: (i) how strongly does the Language of the Question (LoQ) affect factual recall, (ii) does matching LoQ to an entity-associated Language of the Entity (LoE) improve performance, and (iii) do these effects persist when entity popularity is controlled. To this end, we introduce WILA-PopQA, a new Wikidata-grounded benchmark spanning 9 languages with matched popularity profiles, and probe 12 open-weight models of varying sizes and architectures under aligned and misaligned LoQ–LoE conditions. We evaluate models’ answers to 4 types of questions about entity biographical properties in all selected languages. Results show that LoQ is the dominant source of variation. LoQ–LoE alignment does not consistently yield the highest accuracy, and performance depends on the property being asked. These results suggest that prompt language is an actionable experimental factor for multilingual factual evaluation.
2025
Old Reviews, New Aspects: Aspect Based Sentiment Analysis and Entity Typing for Book Reviews with LLMs
Andrea Schimmenti | Stefano De Giorgis | Fabio Vitali | Marieke van Erp
Proceedings of the 5th Conference on Language, Data and Knowledge
Andrea Schimmenti | Stefano De Giorgis | Fabio Vitali | Marieke van Erp
Proceedings of the 5th Conference on Language, Data and Knowledge
This paper faces the problem of the limited availability of datasets for Aspect-Based Sentiment Analysis (ABSA) in the Cultural Heritage domain. Currently, the main datasets for ABSA are product or restaurant reviews. We expand this to book reviews. Our methodology employs an LLM to maintain domain relevance while preserving the linguistic authenticity and natural variations found in genuine reviews. Entity types are annotated through the tool Text2AMR2FRED and evaluated manually. Additionally, we finetuned Llama 3.1 8B as a baseline model that not only performs ABSA, but also performs Entity Typing (ET) with a set of classes from DOLCE foundational ontology, enabling precise categorization of target aspects within book reviews. We present three key contributions as a step forward expanding ABSA: 1) a semi-synthetic set of book reviews, 2) an evaluation of Llama-3-1-Instruct 8B on the ABSA task, and 3) a fine-tuned version of Llama-3-1-Instruct 8B for ABSA.