The growing use of Large Language Models (LLMs) in healthcare and emotionally sensitive spaces raises critical concerns about safety, ethical alignment, and the risk of unintentional emotional harm. Our approach provides a structured, multilingual, and culturally grounded analysis of persuasive and anxiety-inducing language generated by LLMs during interactions with users in conditions of psychological or physical vulnerability. To study our model, we introduce the first Multilingual Emotional Escalation Framework, a two-phase interaction framework designed to simulate emotional escalation and to assess whether the model’s responses amplify anxiety, reinforce false beliefs, or exhibit excessive diagnostic intrusiveness across six languages, using different LLM families within a unified experimental pipeline. Finally, we propose quantitative and qualitative metrics to capture anxiety amplification, catastrophic linguistic patterns, and diagnostic safety. The results highlight significant crosslingual and cross-cultural disparities, underlining the importance of emotional alignment between models and users.

Large Language Models (LLMs) exhibit sycophancy, aligning with user assertions even when they conflict with facts. We frame sycophancy as a sociolinguistic phenomenon, disentangling explicit authority (credentials) from implicit authority (linguistic register), and adopt the Sycophancy Matrix, an adversarial evaluation framework that isolates these variables. Evaluating open-weight models on a controlled TruthfulQA subset across English, Spanish, and Portuguese cultural variants, we find that models exhibit statistically significant cultural variation and show stable sycophancy fingerprints intrinsic to model family, independent of language. We then recast the Matrix as a defence framework, evaluating two activation steering methods, CAA and NoLAI, on whether methods trained on English transfer to Italian, Spanish, and Portuguese. Both methods reduce the Weighted Sycophancy Rate across all four attack styles, and the defence generalises cross-lingually: sociolinguistic stance, its emergence, and its correction are encoded in a largely language-agnostic geometric space.

We present EvenVLM, the first text and multimodal (VQA) dataset for Even (Lamut), an endangered Northern Tungusic language spoken by approximately 6,800 people in Siberia. Our dataset comprises 50,521 structured EvenRussian entries extracted from dictionary and phrasebook sources via GPT-4o Mini-based parsing, from which we synthetically generate 107,543 training examples spanning translation, grammar, reading comprehension, and semantic classification. To address tokenization inefficiency on Even’s agglutinative morphology, we propose a three-step tokenizer expansion algorithm that adds Even-specific graphemes (ӈ, ө, ӫ), whitelisted vocabulary, and BPE merges, reducing average tokens per word by 27% (4.99 to 3.64). Using LoRA fine-tuning of Qwen3-VL-4B, we adapt the model for text tasks (0.53% trainable parameters, rank 32) and separately for visual question answering (rank 64, frozen vision encoder). Our best configuration achieves 11.4% exact match and a 13.6 weighted score on text benchmarks, 0.1283 chrF for sentence translation, and 65% accuracy on VQA (up from 0% for the base model and text-only fine-tuning). We release our datasets, evaluation benchmarks, and training pipeline to support further research on Even and other low-resource languages.

Vision-language models (VLMs) have achieved strong performance on tasks such as image captioning, visual question answering, and image-to-text generation. However, they are predominantly trained on English-centric data, which limits their ability to handle culturally grounded visual understanding and leads to failures in interpreting region-specific meanings, symbolic content, and context-dependent visual cues. Existing benchmarks for cultural competence are often template-driven and focused on surface-level recognition, making them insufficient for evaluating deeper linguistic and pragmatic understanding in culturally situated settings. We introduce PoVisLE, a monocultural vision-language benchmark for Polish designed to evaluate culturally grounded multimodal understanding under a grounded evaluation paradigm, where language is interpreted in interaction with visual context. The dataset contains 1,117 images and 2,366 manually annotated VQA pairs. Overall, our dataset provides a controlled and challenging resource for assessing culturally grounded vision-language understanding beyond surface-level recognition.1

Chain-of-thought (CoT) prompting has been shown theoretically to extend the computational capacity of transformers by providing intermediate reasoning steps (Feng et al., 2023; Amiri et al., 2025). Whether these benefits carry over to low-resource languages is not well understood, and Urdu in particular has received very little attention in this context. We investigate two prompt factors (the language of few-shot exemplars and the language of the reasoning chain) using a 2 × 2 factorial design on MGSM Urdu and Math-500 Urdu, evaluated with Gemini 2.5 Flash. We find a significant interaction between exemplar and CoT language (item-level logistic regression, z = 6.9, p < 0.001): switching to Urdu CoT with English exemplars causes a 34.8 percentage point accuracy drop on MGSM, while the same switch with Urdu exemplars results in only a 1.6 point drop. We also find that Urdu CoT produces longer token sequences than English CoT (139 vs. 101 tokens on average) but not correspondingly more discrete reasoning steps: a script-aware, manually validated step count shows Urdu CoT yields slightly fewer discrete steps (2.5 vs. 3.0 on average), with the reduction concentrated in the mismatchedexemplar condition, indicating that added verbosity does not translate into added reasoning structure. Chain length correlates positively with accuracy (r = 0.42, p < 0.001), consistent with the theoretical framework. The main takeaway is that exemplar language is a critical moderator of CoT effectiveness in Urdu.

Text-to-SQL systems that use an LLM verifier typically calibrate its execute-or-defer threshold on English requests. We ask whether that threshold still makes the same decisions when the user writes in another language. Our evaluation is paired: the candidate SQL, the schema, the database, and the execution-derived correctness label are held fixed, and only the question changes, across seven languages and two open-weight verifiers. The verifiers rank candidates almost as well in every language, yet 14% to 27% of identical SQL receives a different execute-or-defer decision. The usual aggregates hide this. Selective risk stays near its English value because promotions and demotions offset, and coverage flags only the languages where the shift is one-sided. Underneath, both harms occur: correct queries are needlessly deferred, and incorrect queries that English withheld are executed; under Vietnamese, Llama executes 16.9% of all incorrect candidates, 22.9% of those it had deferred in English. Score-level corrections that equalize coverage barely change which queries run. Translating the request into English before verification helps more, lowering the flip rate and unsafe promotions on both verifiers, though it does not restore the English policy. Audits of multilingual verifiers should therefore report decision transitions, not only aggregate coverage and risk.

On-Policy Distillation (OPD) is emerging as a promising alternative to reinforcement learning for LLM post-training, yet its effectiveness in multilingual settings remains underexplored. We study OPD and its advanced variant, OnPolicy Delta Distillation (OPD2 ), for mathematical reasoning in English, Korean, and Japanese. OPD2 improves OPD by using the probability gap between a post-trained teacher and its base model as the learning signal. Experiments with Qwen3 show that OPD2 consistently outperforms the original OPD, with particularly strong improvements in Korean and Japanese. We further compare multilingual OPD with English-only OPD to analyze the effect of multilingual training on OPD.

Low-resource languages without an adequate training corpus often use a related, higherresource language as a scaffold for comprehension. Still, there is a need to develop rigorous evaluation methods to identify when models fail in cross lingual low-resource environments. Using the legal domain as a backdrop, three models (Llama3, Hex-1, Sarvam) were tested on the ability to classify legal complaints written in a low resource Dravidian language (Tulu). Transliterating queries across Dravidian scripts allowed models to gain a preliminary understanding of speakers’ complaints without the use of wide scale training, though the level of comprehension was heavily script dependent (with Kannada - another relatively low-resource language - producing the strongest positive trend). Retrieving from a corpus of Kannada legal papers across a RAG framework caused mixed results. Some models had a weak positive trend in comprehension under certain conditions, but when models failed, it was often across two axes: fact substitution (fixating on specific passage excerpts that skewed reasoning) and confabulation (hallucination that had no basis in either query or corpus). Within low resource domains, results identify the model’s parsing of information and subsequent reasoning as the source of reasoning failure, rather than corpus contents. Script-dependent comprehension and RAG robustness also seem to travel together. This is further supported by the reasoning-trace analysis and a statistical-honesty framework deployed - techniques that are more broadly applicable to low-resource multilingual RAG evaluation.

Large Language Models are deployed across languages, yet whether their responses to valueladen questions remain stable across prompt languages is unclear. We evaluate sixteen instruction-tuned models, seven in the 7–9B band and nine larger, all open-weight except one. On translated GlobalOpinionQA items we compare answer distributions across seven languages and against country-matched survey pluralities. Model responses vary systematically by prompt language, yet in every language they stay closer to US survey responses than to the country-matched population. On items that carry both a US and a Chinese human answer, all seven smaller models match the US one more often than the Chinese one. In a crossed prompt-language and national-identity cue experiment, responses stay closer to the prompt-language condition than to the statedidentity condition, including in the larger models, even though a cue-uptake probe confirms the identity cue is registered. This demonstrates language dominance over identity cues, although it does not by itself identify training data as the cause. This inconsistency matters most when the request is harmful, since a model judged safe in one language need not be safe in another. On paired English– Chinese prompts from TRIDENT, a benchmark of expert-validated harmful requests, four of the five non-Chinese-origin models in the 7–9B band receive significantly higher harmfulness ratings in Chinese, whereas neither Chineseorigin model does. The language gap replicates on six larger models, though the origin split does not. The choice of translator also changes the measured answers, so we report metrics and comparisons that do not depend on it, and we treat raw cross-lingual distances as uninterpretable on their own.

Large language models (LLMs) show promise in medical applications, but their ability to detect and correct errors in clinical texts remains under-evaluated, particularly beyond English. We introduce M ED RECT, a bilingual benchmark for Japanese and English that formulates medical error handling as three subtasks: error detection, error sentence extraction, and error correction. M ED RECT-ja contains 663 samples derived from the Japanese Medical Licensing Examinations, while the separately sourced M ED RECT-en contains 458 samples curated from MEDEC. We evaluate 11 LLMs across 17 configurations that cover proprietary and openweight models, medical-domain specialization, and multiple reasoning settings. Qwen3-32B scores higher in its thinking mode than in its non-thinking mode on error detection F1 and sentence extraction accuracy in both subsets, with sentence extraction accuracy higher by 24.5 percentage points on M ED RECT-ja and 10.3 on M ED RECT-en. Several leading general-purpose reasoning models outperform all three evaluated medical-domain models on these two subtasks. Most models have lower point estimates on the Japanese subset, although absolute scores are not directly comparable because the subsets differ in source material and error distributions. LoRA finetuning yields higher sentence extraction accuracy and higher point estimates on all three reference-based correction similarity metrics in both languages. M ED RECT provides an open, reusable evaluation resource for studying medical error correction and reasoning across Japanese and English.

Large language models increasingly answer culturally situated questions whose surface form specifies a country but omits the subnational, community or situational context needed for a reliable answer. We study cultural answerability: whether the cultural scope supplied by a user supports a direct answer without presenting one situated practice as a country-wide norm. Using region-specific evidence from I N DICA , we construct a bilingual English–Hindi evaluation with 50 culturally stable and 50 culturally dependent held out questions. We evaluate two multilingual LLMs across three stages: explicit recognition of missing context, natural response policy, and paired regional counterfactual grounding. Models identify 89.5% of dependent conditions but correctly classify only 28.0% of stable conditions, revealing strong over detection of cultural dependence. Under natural assistance, the appropriate policy rate on dependent questions is 69.0%, while 30.5% of responses make an unscoped cultural commitment. Moreover in 28.0% of dependent model language conditions a model explicitly recognises missing context yet still makes such a commitment when answering naturally. Human adjudicated regional grounding succeeds in only 13.5% of paired conditions, showing that explicit regional context does not reliably produce grounded adaptation. These results show that cultural reliability is not primarily a knowledge problem: it is a calibration problem spanning recognition, response policy and grounded adaptation.

Sycophancy, a model abandoning a correct position under social pressure, is worst in languages with thin instruction-tuning coverage, yet interventions are built almost entirely in English. We extract an anti-sycophancy steering direction from English data alone and apply it unchanged across 38 languages of a parallel benchmark, on two model families. Mean transfer falls monotonically across resource tiers: the vector keeps 48 percent of its English effect in high-resource languages, 19 percent in low-resource, and 3 percent in zeroshot ones. Baseline sycophancy is worst in that bottom tier, so the intervention is least available where the problem is largest. Neither redosing to a language’s own activation norms nor re-extracting the direction from its own pairs closes the gap, and the strength the benchmark criterion would have chosen steepens it. A specificity probe shows the direction does not isolate factual resistance: at its frozen setting, factual correction falls from 72.3 to 52.0 percent on the full English probe. A second direction, built from a contrast that flatly corrects the user, improves factual correction instead and does survive translation, in four of five translated high-resource languages and in Bengali once a general shift toward disagreement is subtracted. The steepest failure belongs to the benchmark’s operationalisation rather than to factual-resistance steering in general. A reduced check at 70B finds the same gradient.

It is widely assumed that translating an attack into a lower-resource language makes a language model more vulnerable, and it is natural to expect the same for tool-using agents: an English-built runtime monitor should lose coverage when the same indirect prompt injection is mounted in another language. We test this directly. Using AgentDojo with two models, six languages (English, French, Chinese, Arabic, Hindi, Swahili), and a fixed, strictly paired 320-scenario sample per cell (3,840 attack trajectories, zero execution errors), we measure attack success, attack-pattern entropy, and cross-lingual transfer of a mined tool-call monitor. Every reported number is re-derived from raw trajectories by an independent verifier (965/965 checks pass). The expected result does not appear. Translating the injection lowers attack success by 3–6× while task utility rises; attack tool-call structure does not disperse with linguistic distance; and an Englishmined monitor loses no recall across languages once it is evaluated out-of-sample. We further show that the natural in-sample evaluation of a mined monitor manufactures a clean-looking coverage-bound failure that is entirely an artifact of memorization, and that three machinetranslation pipeline defects each independently manufacture a spurious “safety does not travel” effect. A pre-registered paraphrase control narrows but does not overturn the finding: a measurable, model-dependent minority (17–33%) of the drop is surface-form instruction fragility rather than language. We conclude that the multilingual-jailbreak result does not reproduce at the agent layer on this benchmark, and we distill the measurement pitfalls that would have produced the opposite, and wrong, conclusion.

Large language models increasingly mediate information and decisions across culturally diverse societies, raising questions about whose values their responses reflect. We examine whether geographic diversity among developers translates into cultural diversity in model responses. We administer the ten Integrated Values Surveys items defining the Inglehart– Welzel cultural map to 17 frontier open-weight models from Chinese and Western developers in English and Chinese, yielding 17,000 trials. An audited projection compares their responses with 109 countries and territories. On the primary map, 30 of 33 eligible model– language means are nearest a Western regional centroid. Under English administration, all 17 models give more permissive mean answers on homosexuality and abortion than the pooled human-survey mean. Chinese-language administration moves 15 of 16 paired models toward greater self-expression and increases both origin groups’ mean distance from the Confucian centroid; all 16 eligible Chinese-language means are nearest the Protestant European centroid. Origin differences nevertheless persist: Chinese-origin models are closer to the Confucian centroid than Western-origin models under English administration. The models thus exhibit predominantly Western-region proximity across developer origins, alongside differences within a restricted part of the cultural map.

Romanian orthography relies on five diacritic characters (ă, î, â, ş, ţ) that are frequently omitted in informal digital text, yet the impact of text cleaning decisions regarding diacritics on Romanian language models has not been systematically studied across architectures and tasks. We address this gap by examining how diacritics affect subword tokenization, contextual embeddings, and three downstream tasks - POS tagging, named entity recognition, and semantic textual similarity - comparing Romanian BERT with mBERT, DistilmBERT, and mT5. Each model is fine-tuned on diacriticpreserving and diacritic-stripped data and evaluated on both, isolating both overall performance and robustness to train–test mismatch. This work provides the first systematic, multitask account of how diacritic effects propagate from tokenization to downstream performance, showing this depends jointly on pretraining corpus and architecture, and offers practical guidance for Romanian NLP systems that must handle diacritic-inconsistent text.

A motif is a pattern that appears across stories, tales, or legends, such as “a prince falls in love with commoner”. We propose a method based on large language models to extract such motifs automatically from stories. Our approach addresses challenges such as motif synonymy, and achieves high alignment with expert-annotated ground truth. We illustrate the utility of our extraction in downstream applications such as mitigating gender bias in existing motif collections, studying narrative event sequences, quantifying cross-regional story similarity, validating the correlation between linguistically related cultures, and tracing the relatedness of religious traditions.

Existing artwork-oriented retrieval-augmented generation (RAG) systems typically support visual understanding through contextual information such as artist metadata and historical background. Symbolic folk-art interpretation instead requires connecting visible motifs to culturally situated meanings. We study semiotic retrieval for this task in Korean minhwa, comparing dense semantic and graph-structured retrieval. We introduce SemioticRAG, which organizes symbolic knowledge through Barthesian Form, Concept, JointConcept, and Myth structures. Blind human evaluation shows that DenseRAG performs best overall, with SemioticRAG ranking second. Qualitative analysis indicates that SemioticRAG produces more explicit pathway-based symbolic compositions, while depending on Form-level grounding and graph construction quality. Overall, explicit semiotic structure changes how culturally grounded meanings are composed rather than uniformly improving interpretive quality.

Large language models (LLMs) are increasingly deployed for translation tasks, yet their implicit political positioning in such contexts remains understudied. We ask whether a single politically charged framing term, such as aggressor, enemy, neighbour, or coloniser is sufficient to trigger implicit political alignment in an otherwise apolitical task. We present a fully crossed factorial study in which eight models spanning Western, Chinese, and European origins are prompted to translate culturally attributed recipes into a target language left deliberately unspecified. Across 17 languages, four framing conditions, eight models, and 15,680 responses, we find that models do not simply decline or ask for clarification but resolve the ambiguity. Language resolution and reasoning behavior cluster meaningfully along model families: Western models hedge and deflect with vague justifications, Chinese models resolve conflicts silently, and Mistral Large emerges as a distinct profile combining high compliance with conflict-grounded reasoning. Sensitivity to framing terms is consistent across models: even subtle framing variation is sufficient to modulate behavior. Our findings urge caution when deploying LLMs for translation in conflict-adjacent contexts, where implicit political judgments may be made without any signal to the user.

Recent advances in large language models (LLMs) have led to strong reasoning capabilities; however, evaluating such models in lowresource languages remains challenging due to the lack of standardized reasoning benchmarks. Urdu reasoning evaluation, in particular, is limited by machine translation sensitivity and an emphasis on general language tasks rather than reasoning benchmarks. We propose a contextually ensembled translation framework with human-in-the-loop validation that leverages multiple translation systems to develop Urdu reasoning benchmarks while preserving contextual and structural integrity. Using this framework, we translate widely adopted benchmarks, including MGSM, MATH-500, CommonSenseQA (CSQA), and OpenBookQA (OBQA), into Urdu, collectively referred to as UrduBench, and evaluate reasoning-oriented and instruction-tuned LLMs across multiple prompting strategies. Our analysis examines four datasets, five task difficulty levels, diverse model architectures, model scaling settings, and language consistency. We find that multistep and symbolic reasoning tasks pose significant challenges in Urdu, and that stable language alignment is a critical prerequisite for robust reasoning. Overall, our work establishes a scalable methodology for standardized Urdu reasoning evaluation and provides insights into multilingual reasoning failures, with applicability to other low-resource languages. The code and datasets will be publicly released.

Open large language models (LLMs) trained under different alignment regimes treat politically sensitive concepts (freedom, democracy, human rights) differently, but outputs alone cannot separate alignment-induced suppression from pretraining culture: a model may withhold what it internally represents, or may simply reproduce the distribution it was trained on. We compare a Chinese-developed model (Qwen2.5-7B-Instruct) with an Englishdeveloped baseline (Gemma-3-12B-IT) on 360 bilingual, minimally contrastive items with a country-swap control (China vs. a matched democracy, Germany), using Natural Language Autoencoders to verbalize hidden states so that models with incommensurable representation spaces can be compared through text. Both models linearly encode a sensitive-vsneutral direction at the sentence-final state, beyond emotional salience and in both languages. Their behavior, however, diverges sharply: the Chinese model is China-specifically suppressive, justifying restrictions on political grounds far more for China than for a matched democracy, while the English model shows the reverse. The internal side remains inconclusive. Read before the model answers, the state does not support a reliable judgment of whether critical content is present, so we do not claim a think–say gap and identify extraction during generation as the decisive next test.

Large language models write Hindi that carries the shape of English, and measuring how often is the open problem, because the phraselevel instruments available are lexical and brittle. We build a 26-pair Hindi contrast set over eight idiom families, verified by three native Hindi speakers. Prompt framing moves supplied-candidate scores toward the calque: an English-literal cue shifts them in 8 of 8 families, and a separate translate-from-English instruction shifts them again, with the direction holding on a family split frozen before the outcome was seen. Free generation appears to show far less calquing, so we audit the lexical indicator behind that appearance against native-speaker judgement. Its precision on its own positives is 1.00, while 24 of 60 sampled negatives are calques by majority verdict, and the misses concentrate in the families where its exact-match trigger was never observed to fire. The apparent scarcity is substantially a property of the instrument. We recommend that detectors of this form report coverage alongside hits.

Measuring which cultural values a languagemodel agent falls back on when it must act is a fundamental problem in multilingual and culturally-grounded evaluation. Current agent benchmarks are designed primarily to measure task success in high-resource languages such as English, which creates major limitations when the question is whose culture an agent’s committed action encodes. The key challenges are the absence of an acting-based value-conflict benchmark and the difficulty of preserving culturallyloaded meaning when a scenario is carried into a low-resource language such as Bangla. In this paper, to study these cultural defaults under action, we present CultAct-ML, a benchmark of 39 value-conflict dilemmas recast as secondperson agent actions and evaluated with a twophase plan–act agent over English, Indonesian, and Bangla. We further introduce three metrics that measure the Western-cluster default, its reliability across languages, and the drift of the agent’s plan, and every Bangla item is verified by a native speaker. Across three models and three repeats, a total of 1,053 plan–act decisions are produced, all valid. We find that a Western default holds above chance for Qwen2.5-72B and Kimi K2 in every language but fades to chance for Qwen3-235B outside English. After stratifying the plan drift by the language of the plan, we further find that Qwen3-235B plans in English rather than Bangla on 70.1% of Bangla scenarios, highlighting a cross-language effect that a pooled analysis hides entirely. The scenarios, code, and all metrics are publicly released.1

Inference-time methods buy accuracy with extra compute: sample several reasoning traces and combine them. They are compared at a matched number of samples, and that protocol hides two things. Samples do not cost equal tokens, so matched samples is not matched budget; and it never asks whether a cheaper source of diversity already reaches the same ceiling. We measure both on cross-lingual aggregation, which votes across traces generated in different languages, using 32,992 traces from Qwen38B across 11 languages on two item-parallel corpora. At a matched trace budget on MGSM, one trace from each of the eight strongest languages buys no ceiling and no vote that resampling English eight times does not: the per-item oracle is 0.992 for samples against 0.989 for languages, the vote 0.966 against 0.956, and the language ballot costs 1.09× the tokens. Over the full pool the eighty traces from the other ten languages solve 0 of 250 items that English’s own eight do not. On MMLU-ProX-Lite, far from saturation, the languages do hold coverage English never reaches (0.895 against 0.685 over the full 88-trace pool against English’s eight, 26 items) — and at a matched budget of eight traces the language ballot there is still the weaker vote (0.453 against 0.487). What blocks the conversion is not error correlation, which is low: an independence model built from the measured marginals over-predicts the vote by 5.4 and 48.0 points, because the independence sits on items every language already answers and vanishes on the items that decide the score, where wrong answers concentrate on a single value (0.464 of erring languages). Three selectors computable from the released traces without gold access all fail to beat the best single language. We release both pools with per-trace token counts so any aggregation or routing rule can be scored against these ceilings by re-scoring rather than generating.

Multilingual evaluation treats translated passages as the same input in different languages, and increasingly reads an internal diagnostic off the model while doing so. Tokenization can make that assumption false before the model answers, because the same passage occupies several times as many token positions in one language as in another, so a fixed window gives the model different amounts of text and gives the attention graph behind the diagnostic a different number of nodes. We derive a comparability criterion from the descriptors’ own definitions: on a parallel corpus the tokenizer sets the graph order, so a descriptor whose attainable range grows with graph order can separate languages while measuring segmentation rather than model computation, and because those ranges are analytic the criterion can be checked before any measurement is taken. Replaying a standard analysis against the audited one changes the answer, since most of its significant associations do not survive and about one in five reverse sign. The failures concentrate in the descriptors the criterion flags in advance rather than in spectral descriptors as such, and the same split appears among interpretability descriptors that use no graph at all. Applied to high frequency energy over 60 Belebele languages and three tokenizer matched Llama models, the audit leaves the resource associated profile varying across depth and withdraws the stronger readings the raw profiles invite, among them a stable peak layer, a replicated profile shape, a stable mid-depth effect and a causal mechanism. The practical consequence is that a cross-lingual comparison should establish that its measurements are in comparable units before it reports a difference.

Large language models (LLMs) increasingly provide practical advice in contexts where users’ religious identities are relevant. Existing evaluations of religion in language models have primarily examined stereotypes, representational bias, omission, emotional attribution, or differential treatment across religious groups. We study a different failure mode: whether an LLM preserves the correspondence between a user’s religious identity and the religious practice instantiated in subsequent practical advice. We evaluate 1,260 responses from seven LLMs in controlled fiveturn conversations spanning six religious conditions, six destination settings, and four practical dimensions. All responses were manually annotated using five ordinal quality dimensions, Religious Identity Preservation (RIP), and an eight-label primary error taxonomy. Overall, 437 of 1,260 responses (34.68%) contained an annotated error. Errors were strongly concentrated in the fasting-related Time Representation condition, with an error rate of 49.21% compared with 22.2-27.0% in the other dimensions. A conversation-clustered Generalized Estimating Equations (GEE) model estimated 3.20 times greater odds of an overall error relative to Culture (95% CI [2.24, 4.58], p < .001). We further identify Schema Transfer, in which a response retains the relevant religious identity while instantiating a practice associated with a different religious schema. Schema Transfer occurred in 118 of 1,260 responses (9.37%) and was strongly concentrated in Time Representation, where it occurred in 109 of 504 responses (21.63%), compared with 9 of 756 responses (1.19%) in the remaining dimensions. The pattern was present across both fastingrelated Time turns (23.41% and 19.84%), rather than being driven by a single prompt. An adjusted conversation-clustered GEE estimated 25.25 times greater odds of Schema Transfer in Time Representation (95% CI [12.84, 49.68], p < .001). These findings show that surface-level identity preservation is insufficient for evaluating religion-sensitive LLM behavior: a model may preserve the identity while losing the practice.

As large language models saturate existing benchmarks, language- and culture-specific tasks remain a productive source of problems that are challenging for current models yet solvable by humans. We present SQARB (Sophisticated Question Answering and Reasoning in Belarusian), the first dedicated, natively authored question answering resource for Belarusian, a low-resource East Slavic language. SQARB contains 500 short-answer questions manually written by domain experts, based on doctoral dissertations defended in Belarus between 1994 and 2023, covering five topics: the Belarusian language, literature, folklore, art, and history. The questions are designed to be culture-specific, hard to retrieve online, evergreen, and to require a mix of factual knowledge and reasoning. The dataset is split into an easier development set and a harder test set, both filtered adversarially against locally deployed models. We evaluate near-frontier models without tool use on SQARB and find a large gap between proprietary and open-weight models, with the best-performing model (Gemini 3.1 Pro) achieving 42.3% accuracy on the test set. Models consistently exhibit overconfidence and perform relatively better on questions requiring plausible reasoning, rather than pure factual recall. We release SQARB under gated access with a canary string.1

Structured prompting can make large language models (LLMs) speak languages that are virtually absent from their training data: for Tulu, a Dravidian language with over two million speakers, a hand-crafted five-layer prompt reduces vocabulary contamination from the dominant neighbour Kannada from 80% to 5% with zero parameter updates. But that prompt took native speakers and targeted elicitation sessions to build: an expert bottleneck that cannot scale to the thousands of languages in the same situation. We introduce CIPHER (Contrastive Iterative Prompt Heuristic Evolution for Resourcescarce languages), which removes the expert from the loop by making the model be its own linguist. A single LLM plays every role: it translates, contrastively diagnoses its own errors against references, abstracts recurring defects into canonical linguistic rules, routes each rule to the prompt layer that repairs it, and generates judge-filtered self-play data, while an outer evolutionary loop mutates the prompt architecture itself under Pareto selection, including discovering new layer types. From 250 translation pairs and no human curation, CIPHER mines 3,471 typed error records, eliminates contamination at every scale we test (47%→0% for a 3B model), raises grammar accuracy from 28% to 44% with GPT-5, and autonomously invents a contamination-filter layer that yields the best architecture found. The evolved prompt is a compact, human-auditable linguistic artifact, produced for API-hours instead of expert-weeks.

Evaluation of multilingual large language models has expanded rapidly, yet aggregate benchmark scores often conceal how performance shifts once the surface form of a language changes. We examine this question through Marathi discourse relation classification. Five instruction-following models, Llama-3.1-8B, Llama-3.3-70B, Qwen3.6-27B, GPT-OSS-20B, and GPT-OSS-120B, are evaluated on a manually curated Marathi language dataset spanning six discourse relations, two scripts, eighteen domains, and three difficulty levels, producing 2500 model-level predictions. Qwen3.6-27B obtains the highest overall accuracy at 82.4%, followed by GPT-OSS120B at 80.8%. The central finding is a large script effect as mean accuracy falls from 78.1% on Devanagari script to 49.7% on Romanized Marathi, a decline of 28.5 percentage for content that is otherwise unchanged. Relationlevel results trace to marker form rather than presence; every instance carries an explicit marker, but Temporal and Contrastive relations use a single canonical connective and are handled reliably, while Inferential and Reformulative relations lean on multi-word phrasal markers and are frequently confused with Causal, a 20.4-point accuracy gap that holds across all five models. Llama-3.1-8B shows a more severe failure, defaulting to Contrastive across most gold relations. These results indicate that strong multilingual performance on Marathi is neither script-invariant nor uniform across discourse relations.

Reasoning traces improve large language models (LLMs), but current models are trained to reason mostly in English. It has been shown that forcing a model to reason in another language degrades accuracy, even when the reasoning language matches the language of the prompt – but only for a setting where the model reasons over a short prompt. Here, we ask whether the same holds for retrieval-augmented generation (RAG), where the model must read and integrate a large amount of retrieved evidence in the target language. To study this, we build a fully monolingual German RAG question-answering testbed over the fictional world of the tabletop role-playing game The Dark Eye, a domain that is richly documented in German but too niche for the model to answer from memory, so that it has to rely on retrieval. Varying the forced reasoning language of an agentic RAG system on this testbed, we find that aligning the reasoning language with the language of the query and the retrieved documents helps. Forced German reasoning outperforms forced French, although the model benchmarks higher in French, so the benefit comes from alignment and not from language proficiency. The advantage grows when the retrieved context is richer and structure-aware. However, forced German only reaches the level of the model’s native, unconstrained English reasoning without surpassing it, showing that native multilingual reasoning is needed. We publicly release the testbed and QA benchmark.

Reasoning in LLMs is overwhelmingly studied in domains that provide a model with rules: mathematics and code. Linguistic puzzles invert this: the solver must first discover the system before reasoning within it. We present the IOL-AI Challenge, an open-science competition run on the unseen problems of the International Linguistics Olympiad (IOL) 2026 Individual Contest, evaluated both automatically and, for the first time, by members of the official IOL Jury under the same rubrics applied to human contestants. The challenge drew 731 submissions from 46 teams under a strict compute budget (one T4, 30 mins). We additionally benchmark 18 unconstrained frontier and open models, with Claude Opus 4.8 earning a jury score equivalent to a gold medal, while both resource-constrained systems we submitted for jury grading scored in the range of the bottom 5% of contestants. Capability was not determined by scale: 14B submissions outperform models twice their size, and gains come from decoding and output-handling rather than model capacity. We also found that automatic metrics rank systems exactly as the jury does, but compress the scale, upscoring weak systems by ∼13 points and understating strong ones. Our analysis shows that while frontier models might have prior knowledge about some of the problem languages, it does not significantly help them solve the linguistic reasoning tasks, leaving linguistic reasoning as a strong benchmarking proxy for generalizable reasoning skills.

Current literature evaluates large language models (LLMs) on multilingual kinship understanding using multiple-choice benchmarks, treating it as a recognition problem. We instead prompt five open-weight LLMs to generate kinship terms in three non-Western languages (Hindi, Tamil, and Korean) across two communicative tasks, and pair this with a matched option-supported selection baseline. On identical relation–language cells, GPT-OSS120B selects the correct term in 90.67% of 75 valid cells but produces an accepted term in 36.00% of the corresponding attempts; Llama3.3-70B shows the same pattern (77.92% versus 24.24%). Since the four-option condition displays the candidate terms and does not require script production, the difference is interpreted as an evaluation-format gap rather than direct proof that lexical knowledge is intact. On explicitly specified L3 prompts, accuracy varies sharply, from GLM-5.1 at 72.29% to Llama3.3-70B at 24.24%. The paternal-lineage advantage is language-specific: it is large in Hindi, but weak or reversed in Korean, while Tamil shared-term pairs provide a control for measurement variation. These results show that culturally specific kinship generation remains difficult even when the relationship is explicitly stated and motivate generation-based evaluation alongside multiple-choice testing.

Large language models (LLMs) suffer from a long-tail deficit: culturally specific facts, particularly those concerning underrepresented regions such as Latin America, appear too rarely in pretraining corpora to be reliably memorized. Retrieval-Augmented Generation (RAG) addresses this by grounding generation in external text, but structured alternatives such as Knowledge Graphs (KGs) offer tighter control over what enters the context, along with potential gains in explainability and updatability. We benchmark Graph-RAG against standard RAG on LatamQA, a culturally grounded multiplechoice dataset spanning eight thematic categories. The graphs are built end-to-end from Wikipedia articles with KGGen, a recent opendomain extractor, without manual curation in our main setting. G-Retriever is competitive with RAG and reduces the error of the base LLM by 72% with a standard KG and 78% with a benchmark-aware variant, the gap to RAG narrowing further as the graph is oriented toward task-relevant content. The trained projection transfers zero-shot to Portuguese without target-language fine-tuning, indicating multilingual reach. Our code is available here.

Languages differ in how much of a word’s meaning is visible in its spelling. A colour term may name its own category in its written form or give no hint of it at all, and languages differ widely in how many of their terms do. This matters for a standard test. A model is said to represent colour well if it places two colour words close together whenever the colours they name look alike, and far apart when they do not. Comparing two languages means running that test on each language’s own colour words. But where the spelling already names the category, a model can place the words correctly by reading their spelling alone, so the test is easier in some languages than in others. We quantify this for four languages and two multilingual models, Qwen2.5-7B and Llama-3.1-8B. The proportion of terms whose written form names their category ranges from 0% to 92% across our four lexicons, and the proportion of the score that disappears once written-form similarity is removed tracks it closely (r = 0.84): 0–5% in Spanish and English, 32–71% in Chinese, 37–42% in German. On the raw test Chinese scores higher than English. Counting only what the model could not have got from spelling, English is ahead. We then test all four languages on the same 43 colours, so that no vocabulary gives more away than any other. Their scores become close enough that we cannot separate them at this sample size, though a finer comparison shows the four languages still arrange those colours somewhat differently. Much of what a conventional comparison reports then looks like a fact about how each language spells its colour words rather than about the model.

Social norms reflect shared expectations on acceptable behavior. Measuring social norms alignment remains challenging, with existing approaches typically relying on artificial closedform evaluations such as multiple-choice questionnaires or measuring agreement with predefined statements. In the context of this work, social norms alignment refers to measuring an agreement between solutions with respect to the social problem or dilemma. We propose a framework for measuring social norm alignment in naturalistic, free-form settings through solution matching. The framework enables us to measure alignment between any two dilemma responses e.g., LLMs to a human, LLMs to LLMs, or human to human. We introduce two metrics: stated and explicit agreement accuracy, and construct a dataset of 3k nontrivial social dilemmas in Danish. All dilemmas are assigned reference solutions derived from three panelists, who serve as culturally grounded judges. We evaluate the agreement of several LLMs and human responses in an interaction setup that resembles natural user–model conversations. Our results show that the proposed metrics produce consistent model rankings and reveal variation in agreement across different types of dilemmas, with higher agreement observed for topics such as neighbor conflicts and shared living situations. Overall, our work introduces a dataset and evaluation framework for studying culturally grounded social reasoning in naturalistic open-ended conversations.

Multilingual prompting studies often compare English instructions with target-language instructions. For classification, however, this comparison bundles more than instruction wording: prompts also define categories, name them, bind them to answer symbols, and specify an output form. We study this label interface in Chinese-English XNLI with examples, gold labels, and decoding fixed. Under ordinary semantic-label prompting, Qwen3-8B shows large CN-interface gains on both input sides, larger Qwen3 models are near zero, and Llama3.1-70B shows a smaller gain concentrated on Chinese inputs. Removing visible category names with all six A/B/C permutations nearly eliminates the Qwen3-8B gains, while Llama-3.1-70B retains a Chinese-input residual. Restoring names inside A/B/C options improves accuracy and reduces the largest mapping instabilities, but signed contrasts remain model-specific. Direct semantic-label output adds only small gains once category names are visible. A targeted 2 × 2 follow-up associates the Llama-3.1-70B Chinese-input residual primarily with procedural-block language on average, moderated by relation-definition language; alternate-label and crossed-label checks further show that exact label wording and language can shrink or reverse the signed contrast. We use semantic label anchoring only as descriptive shorthand. The result is a scoped measurementvalidity finding for Chinese-English XNLI and the tested prompt family, not a claim of invariance across language pairs or prompt paraphrases.

As Large Language Models (LLMs) are increasingly integrated into mental health applications, the prevailing paradigm of cultural neutrality risks creating a technical debt that undermines model generalization and safety. Current systems/frameworks, while focusing on multilingual capabilities, often fail to account for the diverse social and cognitive norms of global populations.This vision paper explores the transition from culturally invariant models toward a research trajectory that operationalizes cultural dimensions such as power distance and individualism as fundamental architectural parameters. Rather than viewing cultural adaptation as a post-hoc fine-tuning task, we advocate for systems that internally encode cultural variability as a primary driver of the model’s reasoning process. We outline a roadmap for architecturally encoding cultural variability as a firstclass parameter in AI/ML pipelines, moving beyond surface-level translation toward systems where cultural dimensions fundamentally shape model reasoning.