Large language models (LLMs) are increasingly used in culturally sensitive settings, where alignment requires representing diverse preferences within populations. Yet existing methods model populations at coarse demographic or community levels and overlook within-group variation. We introduce DEMOGRAPHIC PLURALISM, an inference-time framework that estimates population-level opinion distributions without opinion-distribution training data or task-specific fine-tuning by generating multiple perspectives within census-grounded demographic groups. Across four backbones on GlobalOpinionQA and VITAL, it reduces Jensen–Shannon distance by 8.4%–26.4% over Modular Pluralism. Among weighted, equal-weighted, and inverse-weighted aggregation, equal weighting performs best overall; group-level error also increases with census weight, helping explain weighted aggregation’s weaker performance.

This study presents a large-scale, exploratory investigation (9,534 trials from 1,159 participants) of color perception, cognition, and emotion in the Korean "Grue" spectrum, finding results consistent with a cognitive advantage for Korean speakers over English-speaking cohorts, particularly in Grue transitions. One illustrative case—the "Australia Divergence"—shows comparable cognitive accuracy between Korean and Australian cohorts in a preliminary, small-sample comparison (132 trials) that nonetheless appears to diverge affectively under the Sentiment-Formality-Friendliness (SFF) framework, consistent with a cognitive-affective decoupling in which perceptual precision may be shared across cultures while affective mapping remains culturally distinct. As a correlational, hypothesis-generating study, we offer these findings as a basis for future confirmatory research rather than as definitive claims, positioning SFF as a promising exploratory tool for capturing culturally mediated "emotional fingerprints" that traditional cognitive metrics may miss, though further validation against human ratings remains needed.

Cultural representation is a prerequisite for the comprehensive evaluation of cultural competence in AI systems. Yet there is no shared, structured way to assess what cultural representation in AI benchmarks involves or how it can be measured. We address this gap by introducing a six-dimensional framework that assesses cultural representation through a structured, dimension-specific scoring instrument, enabling systematic comparison across benchmarks. Applying our framework to 50 benchmarks explicitly targeting culture, we find that none achieves broad coverage across all dimensions. While procedural dimensions such as transparency and bias awareness are comparatively well addressed, benchmarks consistently score lower in linguistic and geographic representation, thematic breadth, and the conceptual grounding of culture. Strikingly, only 32% of benchmarks explicitly define culture, and annotator demographic information is absent in 82% of cases. By making these properties measurable and directly comparable, our framework provides a practical foundation for designing and evaluating benchmarks that more validly assess cultural competence in AI systems.

When annotators disagree on a label, the disagreement itself carries signal—and the number of annotators needed to capture it depends on the evaluation metric. We fine-tune NLI models on label distributions subsampled from ChaosNLI, a dataset providing 100 independent annotator judgments per item, and identify metric-dependent saturation. In our 3-class NLI setting, entropy correlation—whether the model identifies which items elicit disagreement—requires N ≈ 20–50 annotators to converge, while distributional match (KL divergence) saturates by N ≈ 10 (87–95% of improvement across five model seeds); the specific thresholds are estimates for this setting, and it is the phenomenon, not the constants, that we expect to generalize. This finding rests on a prior observation: soft labels carry item-specific signal that label smoothing cannot replicate. Across five smoothing intensities, entropy correlation clusters at r ≈ 0.45–0.49, while soft labels reach r = 0.643 (p < 0.001); per-item analysis traces this gap to smoothing’s inability to distinguish ambiguous items from clear ones. The soft-label advantage replicates across two architectures (DeBERTa, RoBERTa) and a non-NLI-pretrained baseline; differential saturation additionally appears in an exploratory cross-domain evaluation on content safety. These results suggest that annotation budgets should be informed by the target evaluation metric rather than set uniformly.

Detecting the weaponization of cultural heritage on Wikipedia is not a straightforward classification task. Whether an edit manipulates a historical or cultural narrative often depends on context, interpretation, and the perspectives of those assessing it. We examine this challenge through a study of 400 English-language Wikipedia edits concerning Ukraine and Armenia. Three expert annotators independently classified each edit as Potentially Manipulative, Not Problematic, or Ambiguous, before resolving disagreements through structured discussion. We also compared their judgments with those of a large language model. More than one-third of the edits prompted disagreement, and discussion frequently produced decisions that could not be recovered through majority voting alone. The model aligned more closely with the experts’ deliberated decisions than with their initial majority votes, demonstrating that the construction of “gold” labels can substantially affect how model performance is understood. Our findings also show that Ambiguous captures genuine evidential and interpretive uncertainty rather than annotation failure. We argue that culturally sensitive annotation should preserve disagreement, treat ambiguity as a meaningful outcome, and use deliberation, not mechanical aggregation, to construct reference labels.

Large Language Models (LLMs) are increasingly deployed across linguistically and culturally diverse societies, yet evaluations of value alignment remain largely grounded in synthetic benchmarks and normative assumptions. We introduce a population-grounded framework for evaluating multilingual LLM alignment using the European Values Survey (EVS), a large-scale longitudinal study of societal values across Europe. Treating LLMs as survey respondents, we prompt them with EVS questions in multiple European languages and compare model and human response distributions. We analyze correspondence at both country and language-family levels and construct a human value space from EVS responses. Our results show that language families capture a meaningful component of human cross-cultural value variation and provide an informative level of aggregation for multilingual evaluation. The studied LLMs exhibit substantial diversity within this space, occupying distinct regions of the European value landscape rather than converging toward a single value profile. By grounding alignment in observed population responses rather than normative prescriptions, we provide a transparent and reproducible framework for analyzing value-related behavior in multilingual LLMs.

Evaluating language models as student simulators currently focuses on whether their dialogue looks realistic, yet it is unclear whether these models actually respond to the quality of teaching the way human students do. However, this has not deterred prior works from using a model’s uncertainty, measured as the log-likelihood of a fixed correct answer, as a proxy for student behaviour. This approach has been used to simulate student responses, estimate question difficulty, and study “learning". In this work, we test whether this proxy can detect sensitivity to instructional quality. We evaluate five language models on expert-rated tutoring dialogues under tightly controlled conditions. All models reliably distinguish relevant tutoring context from irrelevant context. However, this contextual sensitivity does not extend to pedagogical sensitivity: higher-quality instruction almost never increases the likelihood of the correct answer, and reasoning-oriented models sometimes show the opposite pattern. We argue that this failure is structural, and that exact token-level likelihood is too strict a measure as many different reasoning paths can be correct as opposed to just one fixed correct answer. Our results therefore question the use of target-answer likelihood as a validation metric for LLM-based student simulators and advocate for more robust metrics for learning proxies.

Reward models are central to LLM alignment, yet current benchmarks reveal little about how they represent diverse and sometimes competing social preferences. We extend reward-model evaluation to four socially consequential domains: bias, safety, morality, and ethical reasoning. Our analysis shows that strong benchmark performance can obscure important differences in social behavior. Models may avoid some stereotypes while reinforcing others, and aggregate bias scores can hide these opposing tendencies. We also find that stronger stereotype avoidance can come at the expense of contextual faithfulness, revealing a trade-off between social bias mitigation and sensitivity to context. Across a broad set of reward models, standard benchmark performance is only inconsistently related to socially aligned behavior. These results show that stronger reward models are not uniformly better at navigating pluralistic social expectations and motivate more direct evaluation of the social preferences encoded in reward models.

Alignment may encourage a model to commit to one answer even when human judgments differ. We study this effect by comparing base and aligned checkpoints on ChaosNLI and Measuring Hate Speech using logit, sampled, and verbalized elicitation. Across eight model pairs from 7B to 32B, including a mixture-of-experts model, alignment lowers the entropy of logit-elicited label distributions. Sampled elicitation agrees on the three pairs tested. The effect survives prompt-format, answer-order, word-only, and task-compliance controls. Its implications for human fit are less straightforward: Jensen–Shannon distance worsens on ChaosNLI, but changes in both directions on the native MHS hate-speech task. Six of seven aligned checkpoints in the main sweep fit ChaosNLI annotations worse than a uniform predictor, yet temperature scaling to base mean entropy removes much of the base–aligned gap. Majority-label agreement also changes in both directions. Verbalized probabilities show a different pattern, with entropy increases and decreases across three families and substantial sensitivity to the prompt’s numeric example in some base checkpoints. Staged comparisons of two Ai2 lineages further show that the contributions of SFT and DPO depend on task and prompt format. These results support a consistent sharpening effect under logit and sampled elicitation, while showing why entropy alone cannot establish whether alignment reduces a model’s ability to represent human disagreement.

People hold diverse, sometimes conflicting values, so no single aligned model can satisfy everyone. Pluralistic alignment therefore calls for steerable models that can balance competing objectives differently. Multi-Objective Direct Preference Optimization (MODPO) does this by using an objective weight to span a continuum of trade-offs. We study two questions: when can one model improve two objectives simultaneously, and how can many trade-offs be covered without training a separate model for each? Across seven objective pairs from HelpSteer and UltraFeedback, two pre-training measurements predict whether objectives align or conflict for human-annotated data, but not for AI-annotated data, where response length and repetition confound reward-model scores. For broader trade-off coverage, selecting the nearest trained model and merging model parameters both help, but neither consistently matches direct training. These findings yield practical guidance for building steerable models that serve diverse preferences.

Multilingual red-teaming benchmarks often reduce cultural localization to direct translation, overlooking sociotechnical factors shaping AI safety across contexts. By treating culturally distinct contexts as equivalent, this translation-centered approach can flatten regional diversity, obscure diverse perspectives, and reinforce single-ground-truth assumptions in AI safety evaluation. We present LatAm-RT, a context-aware red-teaming benchmark for incorporating Latin American cultural pluralism into AI safety evaluation. Its Layered Localization Taxonomy (LLT) models localization as an explicit experimental variable across three localization levels direct translation (L1), regional adaptation (L2), and national localization (L3) for Mexico, El Salvador, Costa Rica, and Chile while preserving comparable adversarial intent. We evaluate five language models using 284 prompts across three localization layers, four risk domains, and five adversarial framing strategies. Results show that localization is neither uniformly beneficial nor uniformly harmful: country-specific institutional vocabulary is associated with stronger safeguard activation in Privacy and Civic Systems, whereas politically situated narratives retain their adversarial effectiveness in Political and Information Harm. Adversarial framing remains the primary driver of attack success rate (ASR) variation, including a 55-percentage-point gap between GPT-4o and Claude Sonnet 4.6 under defensive framing. LatAm-RT introduces the Localization Effect (LE) to quantify regional robustness differences obscured by translation-based evaluations. These findings motivate pluralistic AI safety evaluation across culturally situated contexts. The dataset and annotation files are available at https://huggingface.co/datasets/Latam26/LatAmRT

Procedural character generation aims to populate games, simulations, and other virtual worlds with diverse characters. Large language models (LLMs) offer a promising foundation for scaling this task. However, LLM-based procedural character generation remains at an early stage: existing methods either generate characters directly or adapt profiles retrieved from persona banks. As we show, both approaches produce behaviorally homogeneous populations: characters overwhelmingly agree with positive moral norms and respond to questions with helpful, assistant-like reactions. To mitigate this homogenization, we introduce PersonaWeaver, which disentangles world building from behavioral specification and models behavior through setting general, diverse, manually curated banks of moral positions and conversational reactions. This design allows us to test how far LLM(s) can be pushed beyond their default behavioral patterns across settings. Across ten realistic and fantastical settings and three LLM(s), PersonaWeaver produces broader moral and interactional response distributions than prior work. Its guidance also diversifies interpersonal language, response length, and sentiment. It also produces less archetypal combinations of world attributes.

Personalizing jargon detection and explanation is essential for making technical documents accessible to readers with diverse disciplinary backgrounds. However, tailoring models to individual users typically requires substantial annotation efforts and computational resources due to user-specific finetuning. To address this, we present a systematic study of personalized jargon detection, focusing on methods that are both efficient and scalable for real-world deployment. We explore two personalization strategies: (1) lightweight finetuning using Low-Rank Adaptation (LoRA) on open-source models, and (2) personalized prompting, which tailors model behavior at inference time without retraining. To reflect realistic constraints, we also investigate semi-supervised approaches that combine limited annotated data with self-supervised learning from users’ publications. Our personalized LoRA model outperforms GPT-4 with contextual prompting by 21.4% in F1 score and exceeds the best performing oracle baseline by 8.3%. Remarkably, our method achieves comparable performance using only 10% of the annotated training data, demonstrating its practicality for resource-constrained settings. Our study systematically explores efficient, low-resource personalization of jargon detection using open-source language models, offering a practical path toward scalable, user-adaptive NLP systems.

Disagreement-aware NLP challenges the assumption that each instance has a single gold label and raises new questions about evaluation. We examine the evaluation framework introduced in the third edition of the Learning With Disagreement (LeWiDi) shared task, which combines soft evaluation of label distributions with perspectivist evaluation of annotator-specific judgments. We describe the metrics adopted and motivate their adaptation to categorical, ordinal, single-label, and multilabel data. We then analyse their desirable properties and limitations, extending previous work to ordinal and annotator-level evaluation. Finally, using the LeWiDi 2025 submissions, we empirically study the relationship between the soft and perspectivist settings for the submitted systems by distinguishing errors in predicting the overall pattern of labels from errors in assigning those predictions to specific annotators. Our results show that the selected metrics satisfy desiderata while involving important trade-offs. Soft and perspectivist evaluation are closely related but not interchangeable: for most submitted systems, most perspectivist error remains even after predictions are optimally reassigned across annotators, while the additional annotator-assignment error is smaller.

In this work we investigate the sociocultural values learned by large language models (LLMs). We introduce CultureProbe, a novel open-access dataset constructed from natural debate statements using a multi-step methodology. The dataset is synthetically annotated with quantized sociocultural norms according to the Hofstede cultural dimensions. We verify the accuracy of the synthetic annotations using a crowdsourced human annotation experiment conducted on a representative sample. We conduct a comparative analysis between two groups of LLMs developed in different countries (U.S. and China), and use as a comparative baseline patterns observed in human measurements. Using this new dataset and the analysis above, we found that culturally-distinct LLMs largely reflect the values and norms of the countries in which they were developed, highlighting their inability to adapt to the sociocultural backgrounds of their users.

Toxicity judgments can depend on who is judging. We study whether persona prompting captures this variation by evaluating ten Indonesian-capable LLMs under four persona conditions on 7,524 IndoDiscourse posts, each labeled by at least one female and one male annotator, comprising 24,655 individual judgments. Persona cues substantially affect toxicity predictions, but their effects are model-specific, show no common direction, and are small between the two gender personas. Yet this steerability does not amount to perspective adoption: switching between gender personas does not selectively increase label agreement with the corresponding annotator group. Annotation-level mixed-effects models and annotator-level permutation tests likewise find no reliable female–male agreement gap. The apparent gap obtained from aggregated group labels instead follows from annotator label rates and aggregation choices.

This study examines how persona prompting shapes language generated by two multimodal large language models in urban perception, a setting for examining subjective interpretations of shared visual evidence. We organize outputs into three functional layers: descriptive grounding (captions), intermediate semantic layer (perception tags), and interpretive framing (justifications). Using approximately 60,000 persona-conditioned annotations from each of two MLLMs, Qwen3-VL and Gemma4, we find that captions converge strongly across persona profiles and show only small attribute-associated differences. Justifications vary substantially more: economic status produces the largest difference in both models, with political orientation and personality also prominent. Paired image-level comparisons confirm larger justification than caption differences for these three attributes. For perception tags, personas sharing the same attribute level produce more similar tag sets than personas with different attribute levels, with the largest separation observed for economic status. Exploratory topic analysis further suggests persona-specific evaluative emphasis. Across models, profile-pair similarity patterns are strongly correlated for all three output types, although agreement is lowest for justifications. Overall, persona prompting affects interpretive framing more strongly than descriptive grounding.

As chatbots powered by Large Language Models (LLMs) are increasingly deployed across cultural contexts, evaluating whether they can reason about culturally conditioned norms becomes essential. We explore whether the eight workplace-culture dimensions of Erin Meyer’s The Culture Map can provide a practical foundation for such an evaluation. We operationalize these dimensions into 104 behavioral scenarios that test two complementary capabilities: classifying behaviors along a cultural dimension and judging the plausibility of nationality–behavior pairings. We evaluate nine open-weight LLMs spanning 7B to approximately 671B parameters. Nationality information helps some models classify behavior, but most still struggle with group-conditioned plausibility. Explicit cultural rules benefit the reasoning model most, possibly suggesting that this capability depends more on reasoning than on scale. These findings demonstrate both the diagnostic potential of the proposed framework and the limitations that a broader evaluation should capture. Our study is therefore a first step toward a large-scale, open, and multidimensional "Culture Map of Chatbots" benchmark for LLM cross-cultural understanding. Code is available at https://github.com/TH-77-oss/CMC-Benchmark.

Large language models (LLMs) are increasingly used as synthetic survey respondents to estimate population response distributions. In cross-cultural survey simulation, evaluations should assess not only distributional fidelity within countries but also whether differences across countries are preserved. However, existing distance-based metrics such as Jensen–Shannon divergence (JSD) do not directly capture such cross-country differences. To address this limitation, we introduce Cultural Divergence Preservation (CDP), a reference-light diagnostic based on a one-time human calibration. CDP identifies reduced cross-country divergence as cultural flattening and increased divergence as cultural caricature. To evaluate CDP, we conduct experiments across four LLM backbones, three persona-based prompting methods, and two survey domains, the World Values Survey (WVS) and the Big Five Personality Test. The results reveal a systematic discrepancy between conventional fidelity metrics and CDP. Controlled experiments show that CDP changes monotonically as cross-country divergence is attenuated or amplified, while the corresponding changes in JSD remain relatively small. In our audit of real LLM generations, DeepPersona-Inspired prompting is frequently favored by conventional fidelity metrics but exhibits the strongest flattening in every model–domain block. CDP thus complements fidelity metrics by directly quantifying the attenuation or amplification of cross-country divergence.

Misinformation is a growing societal threat, and susceptibility to misinformative claims varies across demographic groups due to differences in underlying beliefs. As Large Language Models (LLMs) are increasingly used to simulate human behaviors, we investigate whether they can simulate demographic misinformation susceptibility, treating beliefs as a primary driving factor. We introduce BeliefSim, a simulation framework that constructs survey-derived, demographic-group belief priors organized using a psychology-informed taxonomy. We study prompt-based conditioning and post-training adaptation, and conduct a multi-fold evaluation using: (i) susceptibility alignment and (ii) counterfactual demographic sensitivity. Across datasets PANDORA and MIST, belief priors improve susceptibility alignment, with larger gains on PANDORA. Finally, post-training adaptation reaches 92.4% accuracy and improves cross-participant generalization.

Large language models (LLMs) exhibit substantial performance disparities between high- and low-resource languages. Beyond lower task performance, they often fail to capture the cultural norms and values of underrepresented communities. In this work, we present the first systematic evaluation of cultural awareness in LLMs for Haitian Creole, a language spoken by millions but severely underrepresented in digital resources. We assess cultural awareness along four complementary dimensions—specificity, bias, diversity, and variation—using a benchmark of culturally salient prompts curated by native speakers in a text infilling setting. Our results reveal a clear gap between cultural awareness in Haitian Creole and higher-resource French, with Haitian performance being more uneven across domains and more affected by French linguistic interference. Story generation further reveals recurring portrayals of Haitian characters through hardship and resilience, showing that even positive characterizations can encode stereotypical narratives. We will make our code, benchmark, and evaluation framework publicly available upon acceptance.

LLMs increasingly serve as general-purpose information systems, including for factual queries about public statistics. In high-stakes domains such as migration, where data accuracy shapes both policy and public attitudes toward at-risk populations, models may answer queries about social groups unequally. We evaluate this concern empirically using GPT-4 Turbo, GPT-4o, GPT-5 chat, and GPT-5.1 on migration-related education statistics. Each model receives zero-shot queries on educational-attainment distributions for national and foreign citizens aged 25–54 across 34 European countries from 2020–2023. We compare outputs against official Eurostat data using RMSE, total variation distance (TVD), cosine similarity, distributional analysis, and response-coverage rates. Across models, results show a consistent gap between plausibility and accuracy: all models preserve broad distributional structure (cosine similarity > 0.93), while deviating substantially from ground-truth values (RMSE 10–14 and TVD 12–16 percentage points). We characterize this combination of structural plausibility and numerical error as an output-level analog of the illusion of competence. GPT-4 models also produce explicit non-answers for 30–78% of responses, with refusals (explicit non-answers) up to 3.4 times more frequent for foreigners than for nationals. GPT-5 models eliminate refusals but return hallucinated values. These findings show that pluralistic evaluation must distinguish legitimate human variation from unsupported model variation and assess whether factual information remains comparably available and reliable across social groups.