Large language models are demonstrating increasing capabilities, excelling at benchmarks once considered very difficult. As their capabilities grow, there is a need for more challenging evaluations that go beyond surface-level linguistic competence. The latter involves not only syntax and semantics but also pragmatics, i.e., understanding situational meaning shaped by context and linguistic and cultural norms. To contribute to this line of research, we introduce SloPragEval and SloPragMega, the first pragmatics understanding benchmarks for Slovene, comprising 405 multiple-choice questions. We discuss the difficulties of translation, describe the campaign to establish a human baseline, and report pilot evaluations with LLMs. Our results indicate that current models have substantially improved in their understanding of nuanced language but may still fail to infer implied speaker meaning in non-literal utterances, especially those that are culture-specific. We also observe a significant gap between proprietary and open-source models. Finally, we argue that benchmarks targeting nuanced language understanding and knowledge of the target culture must be designed with care, preferably constructed from native data, and validated with human responses.

Word associations are one of the most established psycholinguistic methods, offering a long-standing window onto how meaning is organised in the human mental lexicon, yet it remains unclear whether large language models (LLMs) represent and structure associative word meaning in comparable ways, particularly outside English. We present WAX-SL, the first Slovenian dataset of cue–association–explanation triplets annotated with semantic relation labels, and use it to compare human associative behaviour against parallel data generated by three open-weights LLMs. All four sources are annotated with a 20-label relational taxonomy, augmented with a validity marker for output that fails the task constraints. Results show human associations are semantically more diverse and spread across all relation types, drawing more often on action, common phrases, and emotional evaluation, whereas LLM outputs concentrate on structurally transparent relations such as causal and definitional links. The differences are also qualitative: model outputs exhibit instruction violations, semantically incoherent constructions, and hallucinated content. Inter-annotator agreement is markedly lower for machine output (κ = 0.32) than for human data (κ = 0.46), reflecting the greater ambiguity of model responses. Language-specific adaptation does not straightforwardly yield more human-like behaviour. We argue that, on this task, LLMs do not echo human associative behaviour but diverge from it in systematic ways, both in the relations they express and in their markedly higher rate of invalid output.

While new benchmarks for large language models (LLMs) are being developed continuously to catch up with the growing capabilities of new models and AI in general, using and evaluating LLMs in non-English languages remains a poorly-charted landscape. We give a concise overview of recent developments in LLM benchmarking, and then propose a new taxonomy for the categorization of benchmarks that is tailored to multilingual or non-English use scenarios. We further propose a registry of benchmarks implementing the new categorization and documenting benchmarks with a rich set of metadescriptors. While still at a pilot stage, such a registry can lead to a more coordinated development of benchmarks for European languages. We conclude with a review of current trends and advocate for a higher language and culture sensitivity of evaluation methods.