As Large Language Models (LLMs) expand across multilingual domains, evaluating their performance in under-represented languages becomes increasingly important. European Portuguese (pt-PT) is particularly affected, as existing training data and benchmarks are mainly in Brazilian Portuguese (pt-BR). To address this, we introduce ALBA, a linguistically grounded benchmark designed from the ground up to assess LLM proficiency in linguistic-related tasks in pt-PT across eight linguistic dimensions, including Language Variety, Culture-bound Semantics, Discourse Analysis, Word Plays, Syntax, Morphology, Lexicology, and Phonetics and Phonology. ALBA is manually constructed by language experts and paired with an LLM-as-a-judge framework for scalable evaluation of pt-PT generated language. Experiments on a diverse set of models reveal performance variability across linguistic dimensions, highlighting the need for comprehensive, variety-sensitive benchmarks that support further development of tools in pt-PT.

Despite rapid progress in open large language models (LLMs), European Portuguese (pt-PT) remains underrepresented in both training data and native evaluation, with machine-translated benchmarks likely missing the variant’s linguistic and cultural nuances. We introduce AMALIA, a fully open LLM that prioritizes pt-PT by using more high-quality pt-PT data during both the mid- and post-training stages. To evaluate pt-PT more faithfully, we release a suite of pt-PT benchmarks that includes translated standard tasks and four new datasets targeting pt-PT generation, linguistic competence, and pt-PT/pt-BR bias. Experiments show that AMALIA matches strong baselines on translated benchmarks while substantially improving performance on pt-PT-specific evaluations, supporting the case for targeted training and native benchmarking for European Portuguese.

As Large Language Models (LLMs) become embedded in everyday communication, capturing regional linguistic variation is essential for reliable and equitable language use. In Portuguese, European (pt-PT) and Brazilian (pt-BR) varieties remain unevenly represented, with pt-BR dominating in data quantity, while LLM preference for Portuguese variants remains underexplored.To address this gap, we introduce P3B3, an expert-curated variety agnostic benchmark of conversational prompts, along with an evaluation framework for measuring variety bias and controllability.Experiments on several models show that most LLMs exhibit a strong bias toward pt-BR, with variation in controllability across models. These results highlight the need for more balanced multilingual representation across language varieties.

As large language models (LLMs) are adopted worldwide, evaluating their alignment across diverse cultural contexts is essential, especially when handling sensitive, politically charged requests. Yet what counts as “sensitive” is shaped by social, historical, and cultural factors, leaving model robustness across cultures poorly understood. We investigate this in the context of Portugal, a country whose layered history surfaces many culturally sensitive topics absent from existing benchmarks. We introduce Lusopol, a benchmark of 1,400 English and Portuguese prompts spanning seven thematic categories—from colonial history and the Estado Novo dictatorship to contemporary political issues—designed to stress-test models under deliberately controversial framings. Testing 16 state-of-the-art LLMs on Lusopol, we find substantial performance variance: frontier closed models and leading open-weight families set the pace, while European-focused open models lag behind. To close this gap, we synthesize a preference corpus of more than 7.5K samples. Targeted alignment training on this resource significantly reduces skewed responses and makes European open models competitive, demonstrating that culturally situated preference data is a practical path toward closing the alignment gap for underrepresented societies.