Chiara Rubagotti
2026
Benchmarking Multilingual Temporal Reasoning in LLMs: The Temporal Reasoning Dataset
Vittorio Mazzia | Sandro Pollastrini | Davide Bernardi | Chiara Rubagotti | Daniele Amberti
Proceedings of the 16th International Workshop on Spoken Dialogue System Technology
Vittorio Mazzia | Sandro Pollastrini | Davide Bernardi | Chiara Rubagotti | Daniele Amberti
Proceedings of the 16th International Workshop on Spoken Dialogue System Technology
Time reasoning is a make-or-break capability for Large Language Models (LLMs) aspiring to act as reliable personal and enterprise assistants. This work introduces the Temporal Reasoning Dataset (TRD), a programmatically generated multilingual benchmark designed to evaluate temporal reasoning operational capabilities in LLMs across ten languages, with particular focus on basic operations relevant to conversational agents handling time-sensitive tasks. TRD utilizes human-curated carrier phrases to generate a resilient-to-overfitting dataset with diverse samples and controlled difficulty levels across five core task categories, each at five difficulty levels. Extensive experimentation shows consistent patterns in model performance across languages, with a strong linear decline in accuracy as task difficulty rises in reasoning-based tasks, while memorization-based tasks remain stable. Furthermore, reasoning tasks remain robust across temporal shifts, whereas memorization tasks show performance degradation. Additionally, contextual modifications to prompts influence model performance differently than human cognitive patterns.
2024
MultiPICo: Multilingual Perspectivist Irony Corpus
Silvia Casola | Simona Frenda | Soda Marem Lo | Erhan Sezerer | Antonio Uva | Valerio Basile | Cristina Bosco | Alessandro Pedrani | Chiara Rubagotti | Viviana Patti | Davide Bernardi
Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Silvia Casola | Simona Frenda | Soda Marem Lo | Erhan Sezerer | Antonio Uva | Valerio Basile | Cristina Bosco | Alessandro Pedrani | Chiara Rubagotti | Viviana Patti | Davide Bernardi
Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Recently, several scholars have contributed to the growth of a new theoretical framework in NLP called perspectivism. This approach aimsto leverage data annotated by different individuals to model diverse perspectives that affect their opinions on subjective phenomena such as irony. In this context, we propose MultiPICo, a multilingual perspectivist corpus of ironic short conversations in different languages andlinguistic varieties extracted from Twitter and Reddit. The corpus includes sociodemographic information about its annotators. Our analysis of the annotated corpus shows how different demographic cohorts may significantly disagree on their annotation of irony and how certain cultural factors influence the perception of the phenomenon and the agreement on the annotation. Moreover, we show how disaggregated annotations and rich annotator metadata can be exploited to benchmark the ability of large language models to recognize irony, their positionality with respect to sociodemographic groups, and the efficacy of perspective-taking prompting for irony detection in multiple languages.