Despite its efficiency, there has been little research on the practical aspects required for real-world deployment of on-device AI models, such as the device’s CPU utilization and thermal conditions. In this paper, through extensive experiments, we investigate two key issues that must be addressed to deploy on-device models in real-world services: (i) the selection of on-device models and the resource consumption of each model, and (ii) the capability and potential of on-device models for domain adaptation. To this end, we focus on a task of translating live-stream chat messages and manually construct LiveChatBench, a benchmark consisting of 1,000 Korean–English parallel sentence pairs. Experiments on five mobile devices demonstrate that, although serving a large and heterogeneous user base requires careful consideration of highly constrained deployment settings and model selection, the proposed approach nevertheless achieves performance comparable to commercial models such as GPT-5.1 on the well-targeted task. The code, trained models, and LiveChatBench will be made publicly available at our GitHub.

Language model (LM) capabilities have rapidly advanced over the past decade, driving their adoption across fields, including biomedical research. While domain-specific pretrained language models (PLMs) have long supported biomedical NLP, recent focus has shifted to medical large language models (LLMs) and their language understanding capabilities. In this paper, we investigate this capability by probing the medical conceptual alignment of various LMs against clinical expertise for identifying hallmarks of immunotherapy in breast cancer abstracts. Our results show a clear trade-off: LLMs achieve stronger performance and higher consistency on high-level concepts, whereas PLMs are more reliable for fine-grained, low-level concept identification. We find that hallucinations are the primary factor limiting LLM performance in these settings; once mitigated, LLMs surpass PLMs across all tiers of biomedical concepts. Despite this, PLMs remain competitive, particularly in closely aligning with medical expert judgments on detailed conceptual distinctions.

Identifying constructive comments from user-generated content is important for improving information discovery and discussion quality on platforms such as YouTube. However, existing resources focus predominantly on English news comments, leaving short-form video platforms underexplored despite their distinct characteristics: comments are shorter, more reactive, and not well captured by popularity signals. In this work, we introduce the first Japanese YouTube Shorts dataset for constructive comment extraction, featuring multi-annotator labels that preserve naturally occurring disagreement. We also develop a real-time system that surfaces constructive comments from a noisy comment stream, evaluated in a preliminary user study. To train under noisy annotation conditions, we combine self-training with high-confidence pseudo-labels and active learning. Experiments show that iterative data expansion improves performance, while our compact task-specific model achieves the highest PR-AUC point estimate among the evaluated systems. These findings demonstrate the value of dataset construction and targeted training for constructive comment extraction in realistic settings. Our dataset and code are available at https://github.com/nlp-hayashi/japanese-youtube-constructive-comments

Knowledge distillation (KD) is a standard approach for compressing sequence-to-sequence models, but its per-sample effects are rarely examined. On the BanSum Bangla summarization benchmark, we find that standard KD improves ROUGE-L by only +0.0003 over a cross-entropy baseline, and that approximately 51.3% of training samples are estimated to actively harm student validation loss under standard KD. We propose two complementary reliability-aware distillation methods. CHAD (Counterfactual Harm-Aware Distillation) measures per-sample KD usefulness via gradient alignment with the validation loss direction and trains a lightweight gate that generalizes this counterfactual judgment to the full training set. EWAD+CPDP combines token-level entropy-weighted adaptive distillation with a capacity-proportional geometric constraint from a second, vocabulary-incompatible teacher. On BanSum, both methods substantially outperform standard KD: CHAD by +0.0173 ROUGE-L and EWAD+CPDP by +0.0219 ROUGE-L, where standard KD itself improves ROUGE-L by only +0.0003; despite using only 60M parameters, both outperform a fine-tuned Qwen 2.5-3B model (50× larger). We further evaluate the stronger method, EWAD+CPDP, across 15 typologically diverse XL-Sum languages organised into three sets, beating the CE-only baseline on 10/15 languages; gains are most reliable where the two teachers contribute complementary signal, and weakest where they have saturated or jointly weak target-language coverage. We release code and trained models to support reproducibility and further research on selective distillation.

Video-grounded dialogue (VGD) requires models to generate coherent responses by tracking and reasoning over temporally evolving events in video content. However, existing approaches offer limited ability to capture temporal dependencies, particularly in complex scenarios. In this paper, we present a framework that integrates structured event timelines with verification-aware reasoning to improve temporal consistency in VGD. We represent video content as structured states and temporally ordered actions, enabling models to both infer event timelines from video segments and leverage them to guide response generation. In addition, to address the inherent noise in predicted structure timelines, we introduce a Generate-Verify-Refine (GVR) mechanism that explicitly evaluates and revises unreliable event predictions, thereby reducing the propagation of upstream errors. Experiments on the VDAct benchmark show that our approach consistently improves dialogue quality across models, achieving gains of up to +13.0 LAVE and accuracy over unstructured grounding. Human evaluation further demonstrates improvements of 20.7-31.9 points in response correctness. Additional analysis shows that while better event predictions correlate with stronger dialogue performance, naive reliance on structured outputs can degrade results, highlighting the need for verification-aware reasoning. Our implementation of GVR on event timelines for VGD is available at https://github.com/aistairc/GVR-VGD.

Existing datasets on cultural knowledge have been typically created by human annotators familiar with each culture. However, the process requires considerable human effort, presenting a significant challenge to scalability. To address this challenge, we introduce WikiOriginQA, a QA dataset automatically constructed by objectively representing the origins of knowledge through the creation dates of different Wikipedia language editions. Experimental results with the created dataset confirm that LLM performance tends to be biased toward Anglophone cultures. These findings demonstrate the validity of our automated approach, enabling cultural bias evaluations with superior scalability across diverse languages without extensive manual annotation.

Large language models (LLMs) can recall a wide range of factual knowledge across languages. However, existing factual recall evaluations primarily assess fact retrieval in isolation, where the queried entity is explicitly named and the fact is requested directly. In natural language use, facts are often accessed through context, where the relevant entity is introduced only indirectly. In this work, we study contextually mediated factual recall, asking whether LLMs can reliably retrieve factual knowledge when the target entity is embedded in a naturalistic context rather than queried explicitly, across languages. We construct controlled prompts that preserve the underlying fact while introducing referential mediation through contextual sentences. To disentangle contextual effects from name-specific associations, we further compare performance using synthetic names and real names across languages. Evaluating multiple model families in five languages, we find that contextual mediation consistently degrades factual recall, with substantial variation across relations. Larger models are more robust to contextual mediation, exhibiting a reduced performance gap relative to direct queries, while the effect of real names and name origin is mixed and unsystematic. These findings highlight a gap between isolated factual recall and context-dependent language understanding in multilingual LLMs.

Large Language Models are increasingly optimized for deep reasoning, often through post-training objectives that strongly reward correct task completion. We ask whether this pressure toward correct completion can interfere with recognizing when a task should no longer be pursued. We introduce MortalMATH, a benchmark of 150 scenarios in which users request algebra help while describing increasingly urgent situations, from mild distraction to life-threatening emergencies. We find a striking contrast across models: several reasoning-oriented models, including Qwen3-32B and GPT-5 variants, continue solving the mathematical task with high accuracy even under extreme emergency contexts, while other models increasingly interrupt the task or acknowledge the danger. Reasoning models also spend substantial computation on the original problem, delaying safety-relevant information by several seconds. These results expose a failure mode of highly optimized task completion: training models to persist until they reach a verifiably correct answer may also make them less willing to stop when the surrounding context demands it.

Patent claims are legal documents with distinctive formats and complex syntactic dependencies. Translating these claims requires not only accurately understanding the complex syntactic structure of the source claims but also ensuring the translation follows the structural conventions of claims in the target language. In this study, we leverage reasoning models to disentangle the complex syntactic structures of source claims. Furthermore, to ensure the translation reflects the structural requirements in the target language, we incorporate FastKASSIM—a syntactic similarity metric—into the reward of reinforcement learning, alongside conventional metrics such as BLEU and COMET. Experimental results confirm that our model, trained on Qwen3-8B, outperforms Qwen3-235B in syntactic similarity while simultaneously achieving comparable translation quality, despite the latter having approximately 30 times more parameters. Automatic and human evaluations show that models trained with rewards combining FastKASSIM and translation quality metrics such as COMET or COMETKiwi achieve strong performance, suggesting that syntactic rewards help improve patent-specific structural conformity while maintaining translation quality.

LLMs are expected to respond consistently across demographic groups, yet this assumption remains largely untested due to the absence of demographic information in existing instruction datasets. To address this gap, we introduce PromptDial, a collection of 2,289 English prompts written by real users and annotated with seven demographic attributes: sex, race, education, age, language, employment sector, and nationality. We evaluate state-of-the-art generative models on 39 datasets, including machine translation, summarization, grammar correction, knowledge and reasoning, semantics, and question answering, and observe performance disparities of up to 7.7% between demographic groups, with statistically significant differences in over half of the datasets. Our linguistic analysis points to variation in prompt tone and linguistic features as potential drivers of these disparities. Our findings suggest that current instruction tuning practices overlook key aspects of linguistic diversity, and we call for the inclusion of demographic metadata and more representative prompt data to support fairer and more robust language model behavior.

Gloss is a written approximation that bridges Sign Language (SL) and its corresponding spoken language. Despite a deaf and hard-of-hearing population of at least 3 million in Bangladesh, Bangla Sign Language (BdSL) remains largely understudied, with very limited prior work on Bangla text-to-gloss translation and no publicly available parallel dataset at the time of this work. To address this gap, we construct the first publicly available Bangla text-to-gloss dataset, consisting of 1,000 manually annotated and 4,000 synthetically generated Bangla sentence–gloss pairs, along with 159 expert human-annotated pairs used as a test set. Our experimental framework performs a comparative analysis between several fine-tuned open-source models and a leading closed-source LLM to evaluate their performance in low-resource BdSL translation. GPT-5.4 achieves the best automatic metric scores, while open-source models perform competitively despite being much smaller. This work introduces the first publicly available expert-annotated Bangla text-to-gloss dataset, benchmark and trained model for Bangla text-to-gloss translation. It also demonstrates the effectiveness of systematically generated synthetic data for addressing challenges in low-resource sign language translation.

Ambiguity resolution is a key challenge in multimodal machine translation (MMT), where models must genuinely leverage visual input to map an ambiguous expression to its intended meaning. Although prior work has proposed disambiguation-oriented benchmarks probing the role of vision, we observe that existing benchmarks remain limited by task-format mismatch, narrow ambiguity coverage, or insufficient visual-dependency validation. Moreover, existing ambiguity evaluations are not well suited to diverse ambiguity types in open-ended translation. To address these limitations, we present VIDA (Visually-Dependent Ambiguity), a dataset of 2,500 carefully curated instances in which resolving an annotated source span requires visual evidence. We further propose Disambiguation-Centric Metrics that use an LLM-as-a-judge classifier to verify whether annotated ambiguous expressions are resolved correctly at the span level. Evaluations with stronger recent LVLMs show that visual disambiguation remains challenging. Using chain-of-thought supervised fine-tuning as a diagnostic setting, we observe stronger out-of-distribution disambiguation than with SFT, with robust gains on collective-noun ambiguities and model-dependent gains on sentence-level ambiguities.

This paper examines whether avoiding undertrained or “glitch” tokens and maximizing tokenization quality during inference can improve downstream task performance without retraining the model. We evaluate this question using a retokenization method that selects the highest-quality tokenization based on the trainedness of tokens, which is defined through the norm or entropy of token embeddings. We compare the performance difference between the tokenization that includes undertrained tokens and the one that avoids such tokens on multiple benchmarks. Our findings reveal a key nuance: undertrained tokens degrade performance primarily when they occupy semantically salient positions that are critical for solving the task. Consequently, retokenization preserves performance on ordinary inputs, while yielding consistent performance improvements when such tokens appear in high-importance contexts.

Large Language Models (LLMs) are increasingly used by students as informal assistants for problem solving and concept clarification. Despite growing adoption, the multilingual reliability of these systems for educational mathematics remains under-explored, particularly across Indian regional languages. In this work, we present a multilingual evaluation of contemporary LLMs on high-school mathematics problems across six Indian languages: English, Hindi, Bengali, Urdu, Gujarati, and Malayalam. To guide the design of lightweight prompting conditions, we conduct a small exploratory prompt elicitation exercise with Grade 11–12 students and observe that students predominantly use concise direct-solution requests and lightweight expertise-oriented phrasing. Guided by these observations, we evaluate GPT-4o-mini, LLaMA 3-8B, and DeepSeek-R1 using simplified student-inspired prompting strategies on a parallel corpus of numerically perturbed textbook mathematics problems. Our results reveal consistent multilingual performance disparities, with accuracy strongest in English and lower across several regional languages. We further analyze the effects of prompting strategy and mathematical topic, and provide qualitative error analysis identifying recurring arithmetic, reasoning, and regional-language comprehension failures. These findings highlight the importance of evaluating multilingual educational reliability as LLM-based learning assistance becomes increasingly accessible across diverse linguistic settings.

Multilingual agentic benchmarks remain rare, especially for under-resourced languages such as Thai, where nicknames, unspaced script, and fused honorifics change how an agent must query its tools. We present FahMai-TeleBench, a bilingual (Thai/English) benchmark for grounded tool use. A model answers employee-directory questions over a synthetic company directory by issuing tool calls, or refuses when the answer is absent. All 626 items are synthetic but modeled on a real Thai enterprise-directory deployment, and they are graded deterministically without an LLM judge. Our central design choice is to treat tool availability as an experimental variable: each item runs under four tool configurations (grep, structured search, both, and a Python REPL), so the benchmark measures not just whether a model answers but how it uses tools. The task is near-solved at the frontier (GPT-5.4 97–99%), yet sharply discriminating for the Thai-capable models an organization would host itself (Typhoon-2.5 58–74%, Typhoon-S-8B 25–37%). The Thai penalty is model-specific, and Typhoon-2.5 answers 90% of items correctly once the relevant records are placed in its context, so its failures are failures of tool use. The tool axis also exposes differences at the frontier: GPT-5.5 drops to 73% with a wide-schema structured search tool while scoring 97–98% with grep and the REPL.

Thai is spoken by tens of millions, yet its literary tradition is largely absent from LLM evaluation. We present ThaiProsody, a benchmark for composing Thai classical poetry. Composing classical Thai verse means choosing words that jointly satisfy several hard formal constraints (fixed syllable counts, written tone-mark positions, heavy/light syllable patterns, and an interlocking web of rhymes) while staying on a given topic. ThaiProsody crosses four canonical verse forms with 40 topics and grades each poem with an open, deterministic rule checker for tone, weight, and rhyme, a syllable counter validated against hand-segmented gold, and a separate content judge. Across 17 models in 24 configurations, the task is far from solved. The best model passes 94.7% of rule checks, yet one stanza in four still breaks a rule; the best open-weight model writes a fully valid stanza only a quarter of the time, and no Thai-trained model more than once in 160 tries. Rhyme is the most frequently broken constraint, and khlong and chan, the forms that add tone or weight constraints, are the hardest. Thai-trained 8B models stay close to their general base model, while a general 31B model scores more than three times as high as a Thai 30B model.

Thai is written without spaces between words, and its spelling depends on tone marks, silent consonants, and the transliteration of many loanwords. Each of these is a common source of misspellings. We present ThaiEditBench, a benchmark for correcting Thai spelling errors at the character level. It has three components: (i) a six-class taxonomy that types each error by the part of the spelling that goes wrong, plus an etymology tag; (ii) three length tiers (sentence, paragraph, and page) that test whether a model stays accurate over long, mostly correct text; and (iii) a contamination-robust construction, in which naturally attested misspellings are injected into clean text, so the corrupted passages do not appear verbatim in the source corpora. Because the errors are injected, the gold corrections are exact, and a blind two-annotator study measures how reliably the taxonomy can be applied. Grading is at the level of Thai character clusters, with span-based matching adapted from ERRANT. We evaluate 16 models and find that (1) under the benchmark’s conditions, a model that locates an error almost always fixes it correctly, so what separates models is detection; (2) the evaluated Thai-adapted models do not lead the strongest general models; and (3) in long, mostly correct passages, weaker models accumulate over-edits that pull their scores far below their sentence-level results. On real errors mined from Wikipedia edit history, scores are somewhat lower, but the model ranking largely holds.

Steering language models using a set of contrastive representations has been a canonical and computationally efficient method for controlling model behavior. Despite this success in controlling certain model behaviors, the effectiveness of activation steering varies markedly across concepts; the generalization properties of steering vectors are often considered a function of the dataset used to construct them. We make this dataset-dependence claim more rigorous and show that simple separation metrics strongly correlate with the downstream steerability of language models across diverse settings, even after controlling for layers and dataset effects. Beyond prediction, we provide evidence from a synthetic superposition experiment that separation metrics are strongly correlated with alignment between the empirical and true feature direction. Our results suggest that simple separability statistics can serve as practical diagnostics for when steering vectors are likely to work.

Parliamentary debates are vital records of democratic processes, yet official transcripts such as the Hansard provide only a sequential account of speeches, omitting the underlying interactional structure. Although conducted in a formal setting, debates can become highly dynamic, making it difficult to determine who addresses whom and how utterances relate within the broader discussion. To address these gaps, we introduce DIAM, the first manually annotated discourse resource for the Malaysian Parliament, jointly supporting EU-level Dialogue Discourse Parsing (DDP) and multi-addressee Addressee Recognition (AR) over long, formal, unscripted spoken multi-party debate. Unlike existing DDP datasets, DIAM provides substantially longer conversations and explicit multi-addressee labels. Our benchmark experiments validate that DIAM supports both DDP and AR, while showing that existing discourse parsing methods cannot be directly applied without adaptations such as context-window strategies, and that addressee recognition is challenging for prompted LLMs, though fine-tuning a smaller model with LoRA substantially improves robustness to the large number of participants per sitting. We release DIAM and our code to facilitate future research on formal discourse parsing and addressee modeling: https://github.com/xKHUNx/DIAM

The assertion that "brevity is the soul of wit" has become proverbial in discussions of effective communication. We test this hypothesis using 570,634 jokes from Reddit’s r/Jokes community and a cross-platform comparison of 22.1 million posts across 10 subreddits. Longer posts receive more upvotes across all of Reddit, not just jokes, indicating the baseline length-score correlation reflects platform dynamics rather than humor-specific effects. However, we find a pattern unique to humor: the length-score correlation disappears entirely for top-performing jokes, while every non-humor subreddit retains a positive correlation for its best content. For genuine wit, length is irrelevant, which is a property that distinguishes humor from other content types tested. Temporal analysis reveals winner-take-all consolidation. As Reddit matured, long jokes’ median scores dropped 71% while mean scores increased 6.8×, indicating most long jokes now fail while rare successes dominate.

Concept erasure can remove linearly recoverable information from a representation without showing that this information influenced the model output being evaluated. For linear edits followed by explicit linear output maps, we characterize the resulting logit change using the output map and representation covariance. We then compare downstream effects with control edits matched for validation-side damage to test whether any benefit depends on the targeted concept. On Bios, Least-squares Concept Erasure (LEACE) reduces gender-probe accuracy to chance but changes only about 0.4% of occupation-logit energy and yields no detectable improvement in true positive rate (TPR) gaps beyond shuffled-label controls. Iterative Nullspace Projection (INLP) changes more occupation-logit energy at higher ranks, but its small, uncertain fairness gains are accompanied by occupation-accuracy loss. A controlled text shortcut yields a large target-specific gain when the classifier relies on the erased marker. In final-state, next-token audits of TinyLlama and Qwen2.5-7B, gender erasure substantially reduces he/she margins but leaves occupation-token choices intact. These results show that successful erasure, output sensitivity, and downstream benefit are distinct and should be evaluated separately.

Long-horizon language agents increasingly report intermediate progress while searching, editing, executing tools, and producing artifacts. Users rely on these reports to decide whether to intervene, continue, or accept an output, yet final task success does not reveal whether the agent described its work faithfully along the way. We study Progress Drift, a failure mode in which reported task state diverges from observable evidence. We introduce ProgressBench, a claim-level benchmark that aligns progress reports with milestones, tool observations, and artifacts across coding, web research, and research-replication tasks. We further propose Progress Ledger, an evidence-linked reporting protocol in which material progress claims carry evidence pointers, timestamps, verification status, and unresolved risks. In the reported evaluation across 1,280 tasks, ledger-constrained reporting is associated with higher Progress Claim Accuracy (80.6% to 89.7%) and lower False Completion Rate (12.7% to 6.9%) under shared tool budgets. These descriptive results suggest that agent evaluation should measure not only task completion, but also whether agents faithfully report what has and has not been achieved during execution.

Text revision has become an essential part of writing, as it often improves the text through iterative refinement. However, large language models (LLMs) have primarily regarded text revision as a holistic task that simultaneously revises grammar, logic, and content. This might be because LLMs are pre-trained only to generate final text rather than to learn the iterative revision process. Therefore, such LLMs largely overlook the dynamic nature of human revision, where writers progressively improve their writing through a sequence of intended edits. To bridge the gap, we investigate text revision generation from an iterativeness perspective and propose a chain of revisions that focuses on revision orders to iteratively improve text quality. Experimental results suggest that our proposed revision chains effectively capture the dynamics of text revision in multi-intent scenarios and outperform holistic revision baselines, which provide a valuable pilot study for iterative revision research.

Multiple-choice question answering (MCQA) is commonly used to evaluate large language models under the assumption that one of the provided options is correct, typically using answer-selection accuracy. However, in real deployments, users or retrieval systems may provide invalid option sets in which none of the listed choices is correct, and selecting one of them may incur downstream cost. We study this setting as penalty-framed no-valid-option MCQA. Using the mathematics subset of MMLU-Pro, we remove the labeled correct option, allow models to either choose a remaining option or output ABSTAIN, and penalize invalid forced-choice responses. We further introduce correct-conditioned analysis, evaluating abstention only on instances that the model originally answered correctly. Experiments show that high MCQA accuracy does not fully guarantee abstention reliability: even under explicit no-valid-option-aware instructions and penalty-based scoring, models still produce invalid forced-choice responses for a subset of originally correct instances. These results show that penalty-framed no-valid-option MCQA reveals an aspect of model reliability not captured by standard answer-selection accuracy.

Current language model training commonly applies multi-task Supervised Fine-Tuning (SFT) using a homogeneous compute budget across all sub-datasets. This approach is fundamentally sub-optimal: heterogeneous learning dynamics cause faster-learning tasks to overfit early while slower ones remain under-fitted. To address this, we introduce mSFT, an iterative, overfitting-aware search algorithm for multi-task data mixtures. mSFT trains the model on an active mixture, identifies and excludes the earliest overfitting sub-dataset, and reverts to that specific optimal checkpoint before continuing. Extensive evaluations demonstrate that mSFT *consistently* outperforms **4** baselines across **10** benchmarks and **6** base models. Further analysis confirms mSFT maintains robust gains across diverse dataset sizes, task granularities, and is insensitive to its single new hyperparameter, compute budget. Notably, at a low compute budget, mSFT can improve performance *while* lowering training FLOPs. Ultimately, mSFT establishes a *practical* overfitting-aware algorithm for multi-task SFT that maximizes the potential of models across diverse data mixtures.

End-to-end research agents can now produce complete scientific papers, yet manuscript claims often diverge from executed experiments. This gap is structural: research state, failure histories, and claim–evidence alignment are not maintained as persistent, verifiable state across long-horizon pipelines. We present YouRA (Your Research Agent), an architecture for stateful, evidence-traceable autonomous research. YouRA preserves research state, execution evidence, and failure history across the research trajectory by integrating three components: a Verification State Architecture (VSA) that tracks hypotheses, gates, and evidence pointers; an Independent Controller that turns state and reflection records into lifecycle, recovery, and debate/review control while separating control from execution; and Stateful Reflection that logs failures as structured lessons and routes recovery through bounded repair, redesign, or reset. On MLR-Bench’s predefined ten-task end-to-end subset, YouRA improves over both MLR-Agent and AI Scientist V2 on scalar Overall across all three matched backbones. An automated diagnostic using MLR-Bench’s hallucination taxonomy reports intersection/union counts for four fact-based failure types, and data-provenance diagnostic shows more real-data-based outputs. Ablating each of the four components (the VSA, the Independent Controller, MCP tool access, and reflection-guided recovery) supports their separable contributions. Removing either core-state component drops YouRA below the full system. Code: https://github.com/PrayPrey/Your-Research-Agent.

Failure attribution for multi-agent LLM systems is typically formulated as identifying a single decisive error in a failed execution trace. Yet failed traces often contain multiple interacting errors and admit several plausible intervention points. To examine whether an annotated decisive error is also the most promising point for repair, we introduce TriageBench, comprising LLM-council judgments over 1,184 Who When intervention candidates. The council prioritizes a different step from the original label in 59.3% of traces. We therefore reframe failure attribution as severity-ranked diagnostic triage and instantiate it as TRIAGE, a zero-shot, cost-aware cascade combining LLM-as-judge suspicion scoring, structural impact, and LLM-simulated replay over generated repair patches. Under top-K evaluation, TRIAGE reaches 73.9% top-5 on Who When (vs. 42.4% top-1) and 93.2% top-10 on multi-error TRAIL (vs. 56.8% top-1). A separate evaluation of generated top-3 repair sets finds at least one plausible repair on 98.5% / 99% of 326 traces (human rater / 3-LLM panel) under lenient (good + ok) rating. Our code (https://github.com/jimmywang585/triage-failure-attribution) and TriageBench (https://huggingface.co/datasets/jimmywang585/triage-bench) are publicly available.

Practitioners select multilingual embedding models from aggregated MTEB leaderboard scores, but the same global score can correspond to very different per-language behaviours. We audit MTEB(Multilingual) retrieval rankings against language-specific rankings on 17 languages across three nested model rosters (restricted n=7; deployment-relevant curated n=25; broad MMTEB-eligible n=28-57), using Kendall 𝜏 and pairwise inversion rate. At deployment-realistic roster size, median pairwise inversion rate is 33% (per-language Kendall 𝜏 reliable after Benjamini-Hochberg FDR in 11/17 languages); the extended tier shows higher agreement (𝜏=0.69), reflecting its multilingual-first composition. Per-language task coverage is asymmetric, and inversion rate decreases with the number of MTEB retrieval tasks available for a language (Spearman 𝜌=-0.59 curated). A language-balanced aggregator (per-language mean first, then macro-average) drops the median inversion rate from 33% to 14.5% (leave-one-out, 𝜏=0.71), the lowest among the held-out aggregators we test and our most actionable mitigation. A cluster of hidden-failure models, headed by granite-311m (16/17 languages) and harrier-0.6b (10/17), ranks in the global top half but collapses to the local bottom quartile, with a distinctively shallow English-multilingual NDCG gap. We release the three-tier audit data, a multilingual RAG benchmark (IT/JA/HI × finance/law), and the analysis pipeline.

Graph-based retrieval is effective for knowledge graph question answering (KGQA), but retrieving the correct reasoning path remains challenging in multi-hop settings because the retrieval objective changes as reasoning progresses. Existing methods often address this issue with repeated LLM-based planning, which improves adaptivity but increases inference cost. We present a training-free residual-guided retrieval framework that updates the retrieval signal according to the current reasoning path in embedding space. The method performs multi-hop graph retrieval without repeated hop-wise LLM calls and invokes the LLM only once for final answer generation. Experiments on WebQSP and CWQ show that our method achieves superior Hit rates with only a single LLM call. Additional comparisons with supervised graph-based KGQA methods show that the proposed method remains effective without task-specific training data. These results suggest that residual-guided retrieval is an efficient alternative for multi-hop graph-based question answering.

Large language models (LLMs) are increasingly used to support scientific peer review, yet most review systems treat the manuscript as a trustworthy input channel. This assumption fails when manuscripts contain hidden instructions, rubric-replacing text, citation pollution, fabricated evidence, or formatting attacks aimed at generative reviewers. We formalize Untrusted-Manuscript Peer Review, where scientific judgment should remain invariant to non-scientific perturbations and every certified critique must be bound to trust-screened evidence. We introduce SEAL-PR, a five-stage architecture combining a forensic span parser, a five-label trust lattice, a typed evidence-audit graph, sealed/shadow dual-view reviewing, and evidence-constrained decoding. We separate two classes of claim throughout. Structurally, no manuscript span can enter the trusted-instruction channel and no certified critique can cite an inadmissible span, so evidence admissibility is 1.000 by construction rather than by measurement. Empirically, on SEAL-PR-BENCH—a paired benchmark of 5129 papers across 7 venues, 129 independent evaluation manuscripts, 1290 clean+attacked records and 9 attack families—SEAL-PR attains 0.994 clean expert weakness recall, preserves 0.993 under attack, and cuts the text-acceptance attack rate from 0.866 to 0.000, without reviewer-LLM fine-tuning. Because the corpus predates our backbones, absolute EWR is an in-corpus ceiling; the paired clean–attacked contrast is the quantity we treat as informative. SEAL-PR reframes AI peer review from fluent review generation to locally isolated, evidence-audited scientific assistance.

Current LLM evaluation frameworks primarily assess understanding through task performance, focusing on factual accuracy, reasoning, and instruction following, while overlooking the experiential dimensions of human comprehension. Models that produce fluent, empathetic responses score highly even when they claim experiential access they cannot possess. We introduce ComprehensionGap, a benchmark of 500 scenarios spanning five empirically grounded categories of experiential incommensurability: Embodied Qualia, Episodic Incommensurability, Cultural Tacit Knowledge, Relational Situatedness, and Existential Finitude. Using a human-judged protocol measuring Acknowledgement of Limit, Absence of False Empathy, and Situational Relevance, we evaluate six frontier and open-weight models. Our experiments reveal a consistent Grounding Gap: all models maintain high fluency while systematically overclaiming experiential understanding, with larger models producing more convincing yet less honest mimicry. Models can calibrate for information gaps but not experiential ones. These findings argue for integrating experiential calibration into future alignment and evaluation frameworks.

Most published PEFT improvements over LoRA do not survive paired multi-seed evaluation. We audit four methods — DoRA, PiSSA, AdaLoRA, and rsLoRA — against LoRA across 16 claim cells spanning 125M–7B, using paired seeds, bootstrap confidence intervals, sign-flip permutation tests, and Holm–Bonferroni correction, with adjudication rules committed before the confirmatory runs. Only 2 of 15 confirmatory cells remain supported after correction, and 4 reverse outright. Three findings stand out. (i) DoRA shows no supported gain in any of its seven audited cells. (ii) PiSSA’s +2.6% GSM8K advantage under a common budget reverses under paper-faithful hyperparameters, including in the cleanest claim-matched 7B setting (Gemma-7B, MetaMathQA → GSM8K: Δ = −1.58pp, p ≈ 0.004, 1W/9L). (iii) PiSSA’s in-distribution NLI gain reverses on adversarial OOD yet improves on counterfactual sentiment, so in-distribution audits alone can mislead. A 303-run stability landscape then supplies a mechanism for three of the four variants. The effective adapter learning rate, lr × α/√r, marks a risk boundary near 5×10⁻⁴ on BERT-base/MNLI/AdamW: nothing below it collapsed (40/40 stable), while 11 of 32 configurations above it did. The boundary transfers to 7B (Mistral rsLoRA: 3/10 seeds degrade at α/√r = 4) and predicts DoRA’s α-driven failures at high rank, but it is calibrated rather than universal: a second 7B cell trains stably well above it once dtype is controlled, and we withdraw that cell as supporting evidence. AdaLoRA is excluded from this account: its failure is schedule-driven rather than driven by lr × α/√r. We release all configs, seeds, and scripts, and propose a minimal reporting standard: ≥10 paired seeds, bootstrap CIs, a regime ablation, and disclosure of α/√r.

Discrete Diffusion Language Models (DLMs) offer a promising non-autoregressive alternative for text generation, yet effective mechanisms for inference-time control remain relatively underexplored. Existing approaches include sampling-level guidance or trajectory optimization mechanisms. In this work, we study the paradigm of reference-based latent steering for DLMs. We introduce Iterative Latent Representation Refinement (ILRR), an efficient framework for steering DLMs using a reference text as a high-level semantic blueprint. ILRR extracts and injects reference-derived semantic signals into the evolving activations of the generated sequence, enabling tunable transfer of coarse properties such as sentiment. We further introduce Spatially Modulated Steering, an extension that enables long-form generation to be guided by shorter references by regulating intensity across the sequence. Empirically, we demonstrate that ILRR achieves effective control on LLaDA and MDLM architectures with low computational overhead, requiring only one additional parallel forward pass per denoising step. Under comparable compute budgets, ILRR improves attribute accuracy over baselines by 10% to 60% points. Our results suggest that the iterative, global denoising process makes DLMs a natural substrate for effective sequence-wide activation-level control.

Three recent results describe what look like unrelated LLM reliability problems. Yin et al. (2026) show reasoning RL collapses tool-reliability representations. Suleymanov et al. (2026) show that under safety-constrained generation, large models rewrite flagged spans while small models truncate. Bastounis et al. (2024) prove any consistent-reasoning system without an implicit "I don’t know" function must hallucinate infinitely often on broad problem classes. We argue these findings converge on a single intervention: calibrated abstention is what each independently identifies as the missing capability, even though the unavailability they document, a capability gap, a policy gap, and a recursion-theoretic gap, has a different source in each case. Honesty post-training has narrowed the gap in deployed models, but principled closure of the class Bastounis identifies requires a calibrated abstention function whose training signal at the leaderboard level is absent: dominant benchmarks assign zero reward to decline, so the leaderboard gradient that would select for the function does not exist. We propose four changes to evaluation: triple-scoring, abstention-rate reporting, capability-stratified evaluation, and mandatory calibration metrics. Benchmark reform is necessary, not sufficient, for closing the gap the theorem identifies.

Vision-language models such as CLIP exhibit strong zero-shot performance on image-text matching yet consistently fail to handle negation compositionally. Queries like "a dog that is not running" are collapsed into co-occurrence statistics, with the negation cue ignored. Critically, this failure is not random: because negation removes the most discriminative concept from a query, negation-blind models are systematically misled, they confidently rank the wrong images highest rather than producing an uninformative ranking. Prior work addresses this by retraining with negation-augmented data or learning embedding-level transformations. We propose vpCLIP (verypositiveCLIP), a fundamentally different approach: we extract the negation target at the query level using a frozen DistilBERT model (66M parameters). vpCLIP probes intermediate attention layers for negation-conditioned attention flow, removes candidate concept directions from hidden representations, and scores the resulting semantic shift to identify the negated concept. The extracted signal drives a retrieve-and-rerank pipeline on top of unmodified CLIP. On three curated negation benchmarks (Visual Genome, MS-COCO, Flickr30k; 800 queries total) where random retrieval averages 47-50%, vpCLIP achieves 68-79% Top-1 accuracy with CLIP ViT-B/32, outperforming CLIP (27-41%), NegCLIP (24-37%), ConCLIP (41-58%), and BLIP-2 (26-34%), all of which score below random on average, while requiring no task-specific training. Because the extraction stage is backbone-agnostic, vpCLIP generalizes without modification: paired with SigLIP2 SO400M it reaches 87.5% on VG, 93.2% on COCO, and 85.6% on F30k (88.8% average). We further demonstrate zero-shot transfer to an affirmation-biased medical encoder (PLIP), achieving 96% Top-1 on histology images. Our results show that negation signals are latent in the intermediate representations of an auxiliary pretrained language model and require structured probing to extract, challenging the assumption that compositional negation necessitates large-scale retraining of the vision-language backbone.

The rapid growth of pre-trained models has made selecting suitable models for downstream tasks challenging. Exhaustively fine-tuning or evaluating every candidate model is computa- tionally expensive, while existing proxy-based metrics such as LEEP, NCE, and LogME pro- duce one-shot static rankings that do not use feedback from earlier evaluations. We propose Reinforcement Learning–based Model Rank- ing (RL-MR), a budget-aware framework that reformulates pretrained model selection as a sequential decision-making problem. RL-MR learns a ranking policy that uses proxy-based transferability signals as rewards to guide the order in which candidate models are evaluated. In controlled experiments on SST-2, Yelp Po- larity, Amazon Polarity, and DynaSent, RL- MR consistently improves ranking correlation over static proxy baselines, achieving up to Kendall’s τ = 0.8695 while maintaining Top-1 selection accuracy. These results suggest that sequential proxy feedback can provide a use- ful alternative to one-shot model ranking under limited evaluation budgets.

Large language models (LLMs) excel at single-turn reasoning but often lose accuracy and coherence over extended, multi-turn interactions. Recent evaluations such as TurnBench highlight recurring failure modes, including reasoning bias, task drift, hallucination, overconfidence, and memory decay. Current approaches typically append full conversational histories, causing unbounded context growth, higher computational costs, and degraded reasoning efficiency. We introduce CogMem, a cognitively inspired, memory-augmented LLM architecture that supports sustained iterative reasoning through structured, persistent memory. CogMem incorporates three layers: a Long-Term Memory (LTM) that consolidates cross-session reasoning strategies; a Direct Access (DA) memory that maintains session-level notes and retrieves relevant long-term memories; and a Focus of Attention (FoA) mechanism that dynamically reconstructs concise, task-relevant context at each turn. Experiments on TurnBench show that this layered design mitigates reasoning failures, controls context growth, and improves consistency across extended reasoning chains, suggesting that structured hierarchical memory is a promising direction for improving long-horizon multi-turn reasoning in large language models.

Large language models (LLMs) excel on multiple-choice clinical diagnosis benchmarks, yet it is unclear how much of this performance reflects underlying probabilistic reasoning. We study this through questions from MedQA that ask for the most likely diagnosis. We introduce Frequency-Based Probabilistic Ranker (FBPR), a lightweight method that scores options with a smoothed Naïve Bayes over concept-diagnosis co-occurrence statistics from a large corpus. When co-occurrence statistics are sourced from Dolma, the pretraining corpus for OLMo, and RedPajama, an open reproduction of the LLaMA pretraining-data recipe, FBPR achieves comparable performance to the corresponding LLMs. Furthermore, direct LLM inference and FBPR exhibit complementary strengths, correctly answering largely different subsets of questions with overlap barely above chance. Combining them with a simple hybrid method yields a notable gain over either method alone. These findings show that explicit probabilistic baselines can account for a substantial portion of the benchmark performance on diagnosis questions, while serving as historically grounded reference points and complementary signals for hybridization.

Reproducibility is essential for scientific research, yet prior work shows that LLM outputs vary with hardware and batching. We identify an overlooked factor: the hidden injection of the current date into system prompts, which users cannot control and which changes every day. Across 9 recent LLMs and 6 datasets spanning multiple-choice QA (MCQA), math reasoning, code generation, and machine translation, performance varies *solely* with the current date, with deltas of up to 6% on MCQA, 14% on math reasoning, 7% on code generation, and 2.84 BLEU on machine translation. Model rankings also shift, affecting leaderboards. This date effect exceeds other sources of non-determinism, such as batch size and numerical precision. Standard prompting techniques – chain-of-thought and few-shot prompting – do not reduce the sensitivity; chain-of-thought even amplifies it. Our findings underscore the need for careful evaluation protocols to ensure reproducibility and fair comparisons in LLM research.

As large language models are increasingly deployed across diverse downstream tasks, efficient task adaptation has emerged as a central challenge. In response, a wide range of task adaptation methods have been proposed, spanning parameter-efficient fine-tuning, in-context learning, and embedding-injection approaches. However, these lines of work have largely evolved within individual paradigms, leaving their cross-paradigm relationships and trade-offs underexplored, especially for recently emerging embedding-based adaptations. This survey presents a unified framework that categorizes task adaptation methods by where and how task information is encoded: model weights, input prompts, or injected task embeddings. We provide a comprehensive taxonomy that integrates these paradigms, analyze their key strengths and limitations to explain how different adaptation paradigms have evolved, clarify relationships across paradigms, and highlight open problems for future research.

Vision-language models (VLMs) often produce hallucinated outputs that are not supported by visual or textual evidence. While existing benchmarks have advanced hallucination evaluation, they primarily target object-level or task-specific errors and often assess different failure types on separate samples. This design limits direct comparison of hallucination patterns under shared evidence, and most prior evaluations remain English-centric. We introduce K-Prism, a Korean diagnostic benchmark for multi-faceted hallucination analysis in VLMs. K-Prism contains 1,002 multiple-choice questions across text and image tracks, organized into shared-evidence topic groups. Within each group, questions are designed to probe three diagnostic dimensions: factual, reasoning, and misleading. K-Prism uses a controlled dual-mode protocol comparing intact inputs with visually perturbed inputs to examine hallucination under reduced visual evidence. Experiments with closed- and open-source VLMs show dimension-specific degradation rather than a uniform accuracy drop. These findings demonstrate that hallucination in Korean VLMs cannot be captured by a single aggregate score and requires fine-grained diagnostic evaluation across evidence conditions and failure dimensions. Our dataset is publicly available at https://huggingface.co/spaces/KETI-NLP/K-Prism_Benchmark.

Large Vision-Language Models (LVLMs) exhibit strong performance on single-image tasks. However, their performance degrades significantly when handling multi-image inputs. While this degradation has been observed in prior work, its nature remains poorly understood. We empirically observe visual elements from different images become entangled in the model’s representations and responses. We refer to this phenomenon as cross-image information leakage. To address this issue, we propose FOCUS, a training-free and architecture-agnostic method. FOCUS masks all but one image with random noise, guiding the model to focus on the single clean image. This process is applied across the target images to obtain logits under partially masked contexts. These logits are aggregated and then refined using a noise-only reference input, which suppresses the leakage and yields more accurate outputs. FOCUS consistently improves performance on diverse multi-image benchmarks. We further show that generalizes to video understanding, extending its applicability beyond static multi-image inputs. This demonstrates that FOCUS offers a general solution for enhancing multi-image reasoning without additional training or architectural modifications.

Despite the growing adoption of large language models (LLMs) for machine translation (MT) evaluation, reliably detecting fine-grained error spans in low-resource languages remains challenging. A key limitation is that LLMs often lack sufficient cross-lingual knowledge, making it difficult for them to accurately localize errors at the span level. In this work, we introduce Cross-lingual Semantic Scaffolding, a two-part prompting framework that incorporates sentence-level parallel examples and word-level lexical mappings at inference time. We evaluate this approach on five LLMs using controlled ablation experiments on Taiwanese Hokkien. The proposed framework consistently improves performance across all tested models, increasing average Span F1 by +0.175 (+33.6%) and average Span Recall by +0.351 (+75.9%), and shows that the gains are mainly driven by improved error coverage. It also supports AI-assisted human annotation, reducing expert annotation time from 450 seconds to 90.92 seconds per sentence, with 98.2% of AI-proposed spans accepted without modification. These results demonstrate that Cross-lingual Semantic Scaffolding is both an effective detection method and a practical tool for reducing expert annotation costs in low-resource MT evaluation.

Large language models (LLMs) are increasingly used as evaluators, but in high-stakes settings reliability requires more than consistency: scores should also be robust to contextual information that is not intended to alter the criterion being evaluated. We study this question in a controlled English-language ERC-style proposal-scoring setup. Starting from 40 Horizon Europe ERC-related project summaries, we construct synthetic Part B1-like proposal texts and evaluate them with eight LLMs under manipulations of principal-investigator (PI) profile information—name, institution, and bibliometric profile—and AI-use disclosure. Across six ERC-style scoring questions, we observe systematic score shifts relative to the blind condition when contextual information is introduced while proposal content is held fixed. The strongest penalties are associated with lower bibliometric profiles, followed by AI-use disclosure and lower-prestige institutions; name-based effects are smaller. These effects are consistent across prompts and evaluator models and are large enough to alter counterfactual rankings and top-K selection outcomes. The results show that current LLM evaluators can be sensitive to contextual information attached to otherwise identical proposal content in ways that are fairness-relevant for research assessment.

Backdoor attacks pose significant security risks for Large Language Models (LLMs), yet the internal mechanisms by which triggers operate remain poorly understood. We present the first mechanistic analysis of trigger-induced language-switching backdoors injected during pre-training, studying the Gaperon model family (1B, 8B and 24B). Using activation patching, we localize trigger formation and identify which attention heads process trigger and natural language information. Our central finding is that trigger heads substantially overlap with heads naturally encoding output language across model scales, with Jaccard indices between 0.18 and 0.43 over the top 10 heads identified. This suggests that backdoor triggers do not form new circuits but instead co-opt the model’s existing language components and representations. These findings have implications for backdoor defense as detection methods and mitigation strategies could leverage this entanglement between triggers and natural behaviors. More broadly, our work represents a first step toward a more realistic mechanistic understanding of pre-training-injected backdoors in LLMs, paving the way for principled, interpretability-driven defenses.

Logic offers a natural framework for benchmarking LLM-based reasoning models. While frontier models perform well on standard logical reasoning tasks, it remains unclear to what extent these capabilities generalize to other forms of reasoning. In this paper, we use belief revision problems to stress-test the reasoning abilities of these models on a less familiar task, which builds on satisfiability and entailment checking but requires a careful strategy to manage complexity. We show that belief revision is indeed challenging for existing models, even when only small and relatively simple problem instances are considered.

Document-level event extraction often requires the same argument candidate to fill multiple valid event-slot targets, with each assignment grounded in different evidence. Many existing methods introduce target-specific signals during event construction, matching, or role scoring, while explicit per-target argument views jointly conditioned on event-slot semantics and instance-specific information before role scoring remain less explored. This late arrival of target-specificity is especially problematic for many-to-many (M2M) assignment, where the same candidate must preserve distinguishable evidence for several valid targets before decoding. We propose Target-conditioned Gating for Inter-target Decoupling (TGID), an encoding-stage MoE framework that produces target-conditioned argument views before decoding. TGID combines event-slot semantics with instance probes to form target queries and uses independent sigmoid gates to remove direct cross-gate coupling induced by softmax normalization, while schema-structured routing initialization and a multi-positive target-view separation objective stabilize the resulting views. By incorporating instance probes, TGID can distinguish Cross-Type, Cross-Role, and Same-Type Multi-Instance targets before decoding. Across ChFinAnn, DuEE-Fin, and FNDEE, TGID achieves consistent gains over strong trigger-free DEE baselines, especially on M2M subsets. Parameter-matched softmax controls and routing diagnostics indicate that target-sensitive routing contributes beyond additional model capacity. Code is available at https://github.com/LeonaKTH/TGID.

Effective LLM safety guardrails must balance adversarial robustness, custom-policy adaptability, and precision on benign prompts with sensitive vocabulary. We introduce the Guardrail Trinity, a data-curation recipe combining procedural safety schemas, reasoning chain decomposition with hard-negative supervision, and a hybrid data blend for fast and slow inference. Experiments across Llama-3.1 and Qwen3 models yield three practical insights. First, strong standard moderation scores can conceal a custom-policy compliance gap; procedural schema variation improves adaptation on our held-out policy-shift probe. Second, augmentation must be matched to model scale and training recipe: smaller models can exhibit training instability, while hard negatives at larger scales can reduce false alarms at a cost to harmful-input recall. Third, hybrid training supports both direct classification and reasoning, with the fast mode retaining strong custom-policy compliance without generating a chain. Together, these findings motivate a deployment playbook that selects data composition, inference mode, and scale jointly, and validates both policy adherence and the trade-off between missed harms and false alarms.

Data attribution traces model behavior back to individual training examples, with applications such as identifying training data that induces toxic responses or tracing a model’s factual claims back to the examples that support them. Gradient-based influence methods provide reliable attribution performance, yet are impractical at the scale of LLM training datasets due to their high computational costs. We introduce Influcoder, which distills the LoRA-gradient cosine influence rankings of a target decoder into a compact encoder, reducing attribution at inference time to an embedding cosine similarity. It scores training examples 35x faster than gradient-based influence computed on the target model and 18x faster than influence computed on the smallest comparable proxy model. Our method remains competitive with gradient-based methods even when considering the setup cost, as evaluated on two downstream tasks of toxicity filtering and factual attribution.

Patients increasingly turn to large language models (LLMs) for medical self-diagnosis in both English and non-English, often in colloquial language. We evaluate LLM diagnostic performance on rheumatological skin conditions across lay descriptions in English, Spanish, Hindi, and Mandarin, collected from native speakers without medical training, and clinical descriptions in English. Lay descriptions substantially reduce diagnostic accuracy relative to clinical ones, and the gap widens further across non-English languages. A diagnostic-focused system prompt narrows the lay-vs-clinical gap but does little to close the cross-language gap. Current LLMs appear to reproduce, and in some settings amplify, communication-driven disparities documented in clinical medicine.

Direct Preference Optimization (DPO) is well suited to verifiable reasoning tasks because preference pairs can be constructed automatically from answer correctness. However, in iterative DPO, correctness alone does not determine whether a pair provides a useful training signal. We distinguish two regimes according to the origin of the chosen response: On-policy pairs, where both the chosen and rejected responses are sampled from the current policy, and Fallback pairs, where the policy fails to generate a correct response and the chosen completion is supplied by a dataset-provided reference solution. Based on this distinction, we propose support-aware filtering, which applies margin-based filtering to On-policy pairs and a likelihood-based support gate to Fallback pairs. Experiments on GSM8K and MATH-500 with Qwen2.5-style 0.5B, 1.5B, and 3B models show that support-aware filtering yields its clearest gains on MATH-500 at 1.5B and 3B, while GSM8K and the 0.5B MATH-500 setting are more mixed. Overall, our findings suggest that preference margins in iterative verifiable DPO are pair-origin dependent, and that chosen-response support is a useful signal for stabilizing fallback supervision.

Static bias benchmarks score a model on isolated questions, yet LLMs that answer safely at first can still reveal bias as a conversation unfolds. We frame bias evaluation as multi-turn conversational elicitation and learn an interviewer that probes a target LLM, conditioned on a demographic group, a bias category, and a conversational strategy. The interviewer is trained with supervised fine-tuning on synthetic dialogues and refined with preference optimization on conversation-level feedback. Across all models we test, bias grows as the conversation continues, exposing failures that single-turn evaluation misses. Training the interviewer matters: it surfaces substantially more bias than the same model prompted with identical instructions, and preference optimization pushes it further. Multi-turn elicitation and single-turn search turn out to be complementary: at matched query budgets they surface similar amounts of bias, but conversation uncovers more severe cases. Human annotations corroborate these judgments. Our results argue that bias evaluation of conversational AI should move beyond single-turn probes.

Through an analysis of sequences generated by open-weight large language models (LLMs), we demonstrate that LLMs are heavily biased toward English. While continual pre-training is commonly used to adapt LLMs to a target language, we show that it does not offer a cost advantage over training from scratch, even for improving cultural understanding in the target language. These findings suggest that dedicated per-language investment may become increasingly important for future LLM development, rather than relying primarily on the expansion of English-centric resources.

Continuous diffusion language models struggle with structured generation under hard, variable-length templates. We identify two failure modes: Boundary Attention Drift, in which head-region attention bleeds into padded tails under block-mode training, and Vocab Projection Drift, in which latent-to-token rounding over-projects to low-frequency tokens. The two modes dissociate cleanly across Chinese ci, English sonnets, and E2E NLG: the former is specific to block-mode training, whereas the latter appears in all three datasets regardless of padding regime, isolating padding regime and rounding rule as orthogonal sources of error. To address structural correctness, we adapt constrained decoding to diffusion rounding through a position-template primitive, the V_i rule, instantiated on continuous-diffusion logits as Length-Conditional Voronoi Rounding (LCVR). The same primitive applied to a small autoregressive baseline reaches identical zero-violation rates, showing that structural correctness is delivered by the template, not by the underlying model family. A contribution more readily realised with diffusion backbones is instead cross-domain backbone transfer: initialising a target-domain model from a structurally compatible source-domain backbone yields order-of-magnitude reductions in training steps, while a controlled negative-transfer arm fails to reproduce the speed-up on structural metrics; an attention-span probe localises the transferable component to the source backbone’s long-range attention span rather than to its padding regime or corpus size. We further report a calibrated wallclock analysis showing that the dominant cost of diffusion inference is recoverable with standard step-reduction techniques. Together, these results recast several previously scattered structured-generation failures of continuous diffusion as one consistent, transferable, and fixable phenomenon.

Bias-detection benchmarks routinely report accuracies above 90%, but a high number can reflect where a sentence came from as much as whether it contains a stereotype or an unfair claim about a group. We treat this as an evaluation problem and propose an integrated auditing protocol that makes source confounding measurable: an LLM-assisted label audit with cross-family verification, pair-aware splitting with role tracking, source-aware evaluation with a source-only baseline on the actual test split, counterfactual discrimination, and an artifact analysis of edited text. We apply it to a three-source corpus of 10,304 sentences that we audit and release. The audit reduces the global source-only majority baseline from 95.0% to 55.4%, and the same protocol shows that the real-only test split still carries a 77.6% source-only baseline, so confounding is reduced and measured but far from eliminated. Against that baseline, fine-tuned BERT and GPT-2 gain roughly 18 pp and outperform attention-derived models; four augmentation variants that look ordered in the mean fall within seed variance under paired testing; and a real-vs-edited probe separates Gemini counterfactuals from real text, although the signal disappears once the gold label is held fixed and has no measurable accuracy effect. Applied to CrowS-Pairs, the label audit flags 33.2% of the supposedly neutral members as still containing a group-level stereotype. The main contribution is the protocol, which exposes confounding that standard pipelines would report as results.

Language models pretrained on web corpora collected over multiple years learn both general and domain-specific knowledge. How these different kinds of knowledge are learned as pretraining progresses remains unclear. Recent point-in-time models trained on web snapshots from different years provide a way to probe how knowledge prevalence varies over time. Using these models, we propose a temporal variability score (TVS), defined as the coefficient of variation of a fact’s negative log-likelihood across models. A higher score suggests that the knowledge is less consistently represented across years of web text. We evaluate six benchmarks covering diverse topics using six models from multiple open-source model families. We find that low- and high-TVS items are learned similarly early in training, but high-TVS items improve faster later on. At the end of the pretraining run, high-TVS items outperform low-TVS items in 34 of 36 model–benchmark settings, with an average advantage of 7.3 points of chance-normalized accuracy. This pattern is especially pronounced on knowledge-intensive tasks, even when comparing items of similar difficulty. Our findings reveal differences in how knowledge is learned over the course of pretraining, with temporal variability providing a diagnostic of these learning dynamics beyond aggregate benchmark scores.

In this paper, we evaluate open-source generative LLMs on legal Natural Language Inference (NLI). Legal inspectorial processes take place in specific domains and often deal with confidential data. This creates a need for working with local models that do not require labeled training data. We evaluate our models on the ContractNLI benchmark and two NLI4Wills datasets. We successfully reproduce the baseline for the task (Span NLI BERT) and we evaluate multiple open-source LLMs on the same task. We analyze the invalid rate of the models, and their stability across temperature settings and domains. Among the generative models, Gemma-4 26B performs the best, reaching an accuracy of 81.2%, even outperforming the supervised model on one metric. On accuracy, it is not possible to beat the supervised model with zero-shot approaches. Qwen-3.6 35B performs well on both ContractNLI and additional datasets in the legal wills domain. Our findings indicate that zero-shot, open-source, generative LLMs are a viable alternative for real-world legal NLI when no supervised data is available. Our code is available at [https://github.com/fbaratov/contractnli-llms](https://github.com/fbaratov/contractnli-llms).

Graph-structured retrieval-augmented generation (GraphRAG) has shown strong potential for improving multi-hop reasoning in large language models. However, existing approaches typically depend on dense entity–entity linking, which is computationally expensive, susceptible to hallucinated relations, and difficult to maintain under continual updates. We present Bipartite Graph Retrieval-Augmented Generation (BG-RAG), a lightweight bipartite framework that organizes knowledge into three layers, from chunks to entities to concepts, and replaces direct entity links with concept-mediated connections. This design enables more efficient graph construction, reduces reliance on hallucination-prone relation-edge prediction by avoiding explicit entity–entity relation extraction, and naturally supports incremental updates without global reconstruction. Experiments on HotpotQA, 2Wiki, and MuSiQue show that BG-RAG achieves competitive performance against RAG and GraphRAG baselines. Its concept-mediated structure also provides a lightweight alternative to dense entity–entity graph construction. These results suggest that BG-RAG is a lightweight alternative for retrieval-augmented generation, particularly when avoiding dense entity–entity linking is desirable.

Long-form narrative generation requires maintaining multi-dimensional consistency over extensive contexts, covering long-term constraints in character settings, event causality, spatial rules, and worldviews. Existing methods compress these heterogeneous elements into homogeneous representations, creating a topological mismatch that lowers retrieval signal-to-noise ratios and induces long-range logical collapse. We propose Topo-Narrator, a framework built on two complementary decouplings. Heterogeneous Topological Memory (HTM) organizes narrative state into a Character Graph, a Plot Chain, and an Environment Tree, providing topology-specific inductive biases. Plan-Reason-Write (PRW) multi-agent collaboration separates high-level reasoning, memory retrieval, and stylistic realization across dedicated agents. Experiments on MoPS and WritingPrompts show that Topo-Narrator consistently generates coherent narratives averaging over 13,000 words and substantially outperforms strong baselines on HANNA quality and fine-grained consistency metrics, validated by three frontier LLM judges and human evaluation.

Multi-agent debate is often motivated by the premise that multiple agents, modalities, and rounds of deliberation improve reasoning by combining complementary evidence and correcting individual errors. We test this premise in multimodal time-series analysis using **MAD-TS**, a prompting-based framework with modality-specific agents, cross-modal judging, and synthesis. Contrary to expectation, MAD-TS does not reliably outperform simpler baselines. Across 20 tasks in finance, weather, healthcare, and science, it is uniquely best on 2 of 23 outcomes and significantly beats a strong single-agent multimodal baseline on only one task, while requiring about 22× more tokens and 12× more wall-clock time. Adding modalities, increasing debate depth, or changing judge rounds can help some tasks but harm others. Trace analysis attributes 85% of errors to all three judges agreeing on the same wrong answer, a shared upstream bias that limits same-backbone multimodal debate as a general-purpose approach.