Gretino: A Greek and Latin Dataset to Benchmark Retrieval Systems in Classical Languages

Hawau Olamide Toyin, Federico Iezzi, Elia Scapini, Giulio Federico, Giovanni Puccetti


Abstract
Semantic similarity search is a method for exploring large text corpora and retrieving conceptually related content. Although widely used in modern language applications, it remains underexplored in the context of classical literature, where it could provide scholars with tools to uncover meaningful connections across authors, genres, and languages, surpassing the limitations of rule-based or keyword search systems. To promote the adoption of semantic retrieval in classical languages, we introduce Gretino, the first benchmark dataset for evaluating semantic search systems in Latin, Ancient Greek, and cross-lingual settings. Gretino comprises 240 carefully designed queries, each paired with five semantically relevant passages in Latin and Greek. The dataset is divided into two subsets: Gretino Silver, consisting of 200 queries and 1,000 targets (evenly split between Latin and Greek), generated with the assistance of ChatGPT and subsequently reviewed; and Gretino Gold, a manually curated high-quality subset of 40 queries and 200 targets, fully based on authentic classical texts. We evaluate four pre-trained language models: GreBERTa, LaBERTa, PhilBERTA, and SPhilBERTa and demonstrate the potential of a contrastive learning approach based on SimCSE (Gao et al., 2021) for fine-tuning, showing that training on carefully curated bilingual corpora, with texts aligned in the two languages, can improve retrieval performance.
Anthology ID:
2026.lrec-1.70
Volume:
Proceedings of the Fifteenth Language Resources and Evaluation Conference
Month:
May
Year:
2026
Address:
Palma de Mallorca, Spain
Editors:
Stelios Piperidis, Núria Bel, Henk van den Heuvel, Nancy Ide, Simon Krek, Antonio Toral
Venue:
LREC
SIG:
Publisher:
ELRA Language Resource Association
Note:
Pages:
919–928
Language:
External URL:
https://lrec.elra.info/lrec2026-main-070
DOI:
10.63317/3ipryhrqmwvi
Bibkey:
Cite (ACL):
Hawau Olamide Toyin, Federico Iezzi, Elia Scapini, Giulio Federico, and Giovanni Puccetti. 2026. Gretino: A Greek and Latin Dataset to Benchmark Retrieval Systems in Classical Languages. In Proceedings of the Fifteenth Language Resources and Evaluation Conference, pages 919–928, Palma de Mallorca, Spain. ELRA Language Resource Association.
Cite (Informal):
Gretino: A Greek and Latin Dataset to Benchmark Retrieval Systems in Classical Languages (Toyin et al., LREC 2026)
Copy Citation: