Marili Tomingas

Author directory

2026

We introduce SMUGRI-COPA, a commonsense reasoning benchmark for two heavily under-resourced Finnic languages: Võro and Livonian. Existing evaluation resources for these languages rely on a translated subset of FLORES sentences, limiting linguistic diversity and the evaluation of broader language understanding. To address this gap, we human-translate the COPA (Choice of Plausible Alternatives) benchmark into Võro and Livonian, creating, to our knowledge, the largest discriminative benchmark currently available for these languages. We evaluate a wide range of proprietary and open-weight language models on SMUGRI-COPA and compare the results with Estonian XCOPA. Our results reveal a substantial performance gap between Estonian and the lower-resource languages, with most models failing to significantly outperform random guessing on Livonian. We further find that enabling reasoning for GPT-5 substantially improves accuracy for Võro and Livonian, while its reasoning summaries show frequent reliance on explicit translation of the tasks. Our work highlights the challenges of evaluating endangered language understanding and provides a new resource for multilingual evaluation research.

2022

Livonian is one of the most endangered languages in Europe with just a tiny handful of speakers and virtually no publicly available corpora. In this paper we tackle the task of developing neural machine translation (NMT) between Livonian and English, with a two-fold aim: on one hand, preserving the language and on the other – enabling access to Livonian folklore, lifestories and other textual intangible heritage as well as making it easier to create further parallel corpora. We rely on Livonian’s linguistic similarity to Estonian and Latvian and collect parallel and monolingual data for the four languages for translation experiments. We combine different low-resource NMT techniques like zero-shot translation, cross-lingual transfer and synthetic data creation to reach the highest possible translation quality as well as to find which base languages are empirically more helpful for transfer to Livonian. The resulting NMT systems and the collected monolingual and parallel data, including a manually translated and verified translation benchmark, are publicly released via OPUS and Huggingface repositories.