Accounting for disagreement among annotators is important for a variety of NLP applications. However, understanding what insights can be drawn from disaggregated annotations remains an open question. We introduce FLOCKE, a framework that jointly learns a task and models latent groups of annotators with similar behavior. We first formalize six analyses enabled by our framework, including annotator-group discovery, group- and individual-level label prediction, and finding the items most prone to cause disagreement. We then validate FLOCKE with a controlled synthetic benchmark, creating predefined annotator groups and labeling patterns, and showing that it can recover group structure with very high accuracy. We apply FLOCKE to two real-world datasets, where it predicts individual annotations, discovers interpretable annotator groups, and shows that groups are not explained by socio-demographic traits alone.

Despite the remarkable success recently obtained by Large Language Models, a significant gap in performance still exists when dealing with low-resource languages which are often poorly supported by off-the-shelf models. In this work we focus on Fassa Ladin, a Rhaeto-Romance linguistic variety spoken by less than ten thousand people in the Dolomitic regions, and set to build the first bidirectional Machine Translation system supporting Italian, English, and Fassa Ladin. To this end, we collected a small though representative corpus compounding 1135 parallel sentences in these three languages, and spanning five domains. We evaluated several models including the open (Meta AI’s No Language Left Behind, NLLB-200) and commercial (OpenAI’s gpt-4o) state-of-the-art, and indeed found that both obtain unsatisfactory performance. We therefore proceeded to finetune the NLLB-200 model on the data collected, using different approaches. We report a comparative analysis of the results obtained, showing that 1) jointly training for multilingual translation (Ladin-Italian and Ladin-English) significantly improves the performance, and 2) knowledge-transfer is highly effective (e.g., leveraging similarities between Ladin and Friulian), highlighting the importance of targeted data collection and model adaptation in the context of low-resource/endangered languages for which little textual data is available.