This paper presents the participation of the L3IRIT team in SemEval Task 4.The team is a joint research group working on narrative extraction from historical text, led by the IRIT laboratory (University of Toulouse) and the L3i laboratory (University of La Rochelle). Our participation is grounded in the construction of a novel bilingual resource extracted from Wikipedia by automatically aligning film plots. Leveraging this dataset, we train embedding models using contrastive learning objectives to capture higher-level narrative structures more effectively. The resulting resource goes beyond surface-level lexical overlap, providing supervision for narrative similarity without manual annotation. In addition, we introduce a named-entity masking strategy designed to promote narrative abstraction and reduce superficial entity-based matching. Overall, our approach aims to support representation learning that captures structural and event-level similarities across stories in different languages more effectively.Our system ranked in 24 of the 44 scoreboards for Task A and 20 of the 27 scoreboards for Task B, achieving accuracies of 65.75 and 61.00, respectively.

Named Entity Recognition (NER) and Named Entity Linking (NEL) are core tasks in entity extraction, yet their robustness is limited when applied to noisy documents, such as those generated by Optical Character Recognition (OCR) over historical documents. Although large language models (LLMs) have shown strong zero-shot and few-shot performance on NER and NEL tasks, prior work has largely focused on using LLMs as direct predictors rather than evaluating extraction performance. In this study, we explore the feasibility of using LLMs as learned evaluators to estimate the quality of NER/NEL outputs, especially in settings where human-annotated references are unavailable at inference time. We propose supervised approaches that fine-tune LLMs to predict quality scores based on training data with gold annotations, enabling reference-free quality estimation once trained. Experiments on the HIPE-2020 benchmark across English, French, and German languages demonstrate that fine-tuned LLMs provide reliable estimates of output quality. Our findings suggest that LLM-based evaluation can support quality control and enable evaluation in noisy setting.

Large language models (LLMs) have recently shown promise for historical entity linking, but preference optimization for this task is often formulated with only one negative candidate per training instance. This discards information from the remaining candidates retrieved for the same mention. We introduce multi-negative direct preference optimisation (MDPO), a reference-based pairwise objective that compares the correct entity with all valid rejected candidates associated with each mention. MDPO preserves the Bradley-Terry formulation of DPO while exploiting the complete candidate set through masked, length-normalised sequence scores. We evaluate MDPO on hipe-2020 and newseye, covering French, German, English, Swedish, and Finnish historical newspaper text. Experiments show that MDPO improves over supervised fine-tuning and single-negative DPO, with particularly strong gains for NIL mentions, semantic ambiguity, OCR noise, and historically difficult names. Further analyses disentangle candidate-generation and selection errors, showing that candidate retrieval remains a key bottleneck for end-to-end entity linking. These results demonstrate that incorporating all within-instance negative candidates is a simple and effective improvement for LLM-based historical entity linking.

Narrative NLP encompasses a diverse set of competencies such as detecting whether a text contains a narrative, classifying its genre or function, completing its plot, and judging similarity between stories. However, the relationship between model capabilities across these sub-tasks remains poorly understood and no unified benchmark exists to evaluate large language models across these dimensions simultaneously. In this paper, we introduce NARRATOR, a unified benchmark of 96,818 multiple-choice questions derived from six narrative datasets spanning detection, classification, story completion, and similarity. Evaluating 50 open-weight LLMs in a zero-shot setting reveals that LLM performance profiles are highly heterogeneous and no model dominates across all sub-tasks. To exploit this complementarity efficiently, we learn compact narrative competence embeddings of LLMs using a matrix factorisation over a binary correctness matrix, without accessing model weights. A single linear layer on top of these frozen embeddings supports model routing (71.1%), correctness forecasting (65.3%), and benchmark accuracy prediction (mean Kendall’s 𝜏 = 0.72) at near-zero inference cost.

Les modèles de langage de grande taille (LLM) sont exploités depuis plusieurs années maintenant, obtenant des performances de pointe dans la reconnaissance d’entités à partir de documents modernes. Depuis quelques mois, l’agent conversationnel ChatGPT a suscité beaucoup d’intérêt auprès de la communauté scientifique et du grand public en raison de sa capacité à générer des réponses plausibles. Dans cet article, nous explorons cette compétence à travers la tâche de reconnaissance et de classification d’entités nommées (NERC) dans des sources primaires (des journaux historiques et des commentaires classiques) d’une manière zero-shot et en la comparant avec les systèmes de pointe basés sur des modèles de langage. Nos résultats indiquent plusieurs lacunes dans l’identification des entités dans le texte historique, qui concernant la cohérence des guidelines d’annotation des entités, la complexité des entités et du changement de code et la spécificité du prompt. De plus, comme prévu, l’inaccessibilité des archives historiques a également un impact sur les performances de ChatGPT.

Dans cet article, nous abordons la reconnaissance d’entités nommées dans des documents historiques multilingues. Cette tâche présente des multiples défis tels que les erreurs générées suite à la numérisa- tion et la reconnaissance optique des caractères de ces documents. En outre, les documents historiques posent un autre défi puisque leurs collections sont distribuées sur une période de temps assez longue et suivent éventuellement plusieurs conventions orthographiques qui évoluent au fil du temps. Nous explorons, dans ce travail, l’idée d’injecter des connaissance temporelles à l’aide de graphes pour une reconnaissance d’entités nommées plus performante. Plus précisément, nous récupérons des contextes supplémentaires, sémantiquement pertinents, en explorant les informations temporelles fournies par les collections historiques et nous les incluons en tant que représentations mises en commun dans un modèle NER basé sur un transformeur. Nous expérimentons avec deux collections récentes en anglais, français et allemand, composées de journaux historiques (19C-20C) et de commentaires classiques (19C). Les résultats montrent l’efficacité de l’injection de connaissances temporelles dans des ensembles de données, des langues et des types d’entités différents.

This paper tackles the task of named entity recognition (NER) applied to digitized historical texts obtained from processing digital images of newspapers using optical character recognition (OCR) techniques. We argue that the main challenge for this task is that the OCR process leads to misspellings and linguistic errors in the output text. Moreover, historical variations can be present in aged documents, which can impact the performance of the NER process. We conduct a comparative evaluation on two historical datasets in German and French against previous state-of-the-art models, and we propose a model based on a hierarchical stack of Transformers to approach the NER task for historical data. Our findings show that the proposed model clearly improves the results on both historical datasets, and does not degrade the results for modern datasets.