Progress in question answering (QA) over tabular data has enabled reliable factual retrieval from relational tables. However, many real-world questions are probabilistic, requiring reasoning under uncertainty and latent conditional dependencies that are not directly stored in individual cells. We introduce LUCARIO, a large-scale benchmark for probabilistic QA over real-world tabular datasets, covering diverse domains, dependency patterns, and natural language variations. We propose Auto-BN, a fully automatic neuro-symbolic framework that induces a Bayesian Network from raw tables, translates natural-language questions into formal probabilistic queries, and performs exact inference. Experiments across multiple LLM backbones show that Auto-BN consistently outperforms NL2SQL, retrieval-based, and premise-based baselines, while maintaining a near-zero error rate. LUCARIO fills a relevant gap and provides a practical and scalable testbed for advancing probabilistic reasoning over structured data.

To protect patient privacy, medical models are frequently trained on synthetic datasets. Focusing on 13 clinical attributes commonly found in breast cancer records, we propose a synthetic data generation framework in which each generated sample is a data–report pair: the structured data capture the 13 attributes of interest, while the accompanying (French) text simulates a corresponding breast cancer pathology report. Data privacy and coverage are obtained by generating the structured data using a tabular data synthesiser. In this work, we focus on enhancing the diversity and quality of the generated texts through a combination of data-driven knowledge distillation, data filtering and data combination. Our findings suggest that the quality of the training data used for the student model outweighs its quantity; that combining data from multiple teachers helps improve performance; and that a compact model (Mistral 7B) can match the zero-shot Information Extraction performance of a significantly larger model (MedGemma 27B) when fine-tuned on high-quality synthetic data.

Large language model-based multi-agent systems (LLM-MAS) are designed through roles, debate protocols, and aggregation rules. These choices create implicit expectations of trust, skepticism, deference, or collaboration. We make inter-agent relations explicit as signed pairwise priors, rendered in natural language and added to system prompts while keeping the task protocol fixed. Across commons governance and multi-agent debate, these priors change how readily agents coordinate or agree, a pattern we call convergence pressure. More positive relations generally improve sustainability within the GovSim relational-prior sweep and increase consensus on subjective questions. These improvements over negative relations differ from gains over no-prior prompting. On objective QA, fully positive priors usually yield lower final-answer accuracy than the no-prior baseline, and some conditions produce more frequent but less accurate consensus. Effects depend on model backbone, relation type, and topology; explicitly neutral relations also produce different outcomes from omitting relational framing. Relational priors are therefore task-specific interventions and diagnostic probes of sensitivity to social framing. Evaluate them against a no-prior baseline using the task’s primary metric, report accuracy and consensus correctness alongside agreement on objective tasks, and keep no-prior prompting as the default for accuracy-centric tasks unless validation supports a relational prior.

Tracking semantic change in low-resource languages across extensive historical timelines presents significant challenges due to data scarcity and the limitations of static embedding alignments. This study investigates the diachronic evolution of the Sinhala language from the 13th to the 20th century using a multi-stage computational framework. We first align century-specific Word2Vec and FastText embeddings using Similarity Matrix Based Alignment (SMA) and Orthogonal Procrustes (OP) techniques, finding that OP alignment provides more stable neighbourhood tracking for identifying temporal similarity dips. To move beyond aggregate measures, we introduce a Bidirectional Semantic Impact Pruning approach using contextualised embeddings from a fine-tuned Llama-3.1-8B. By applying Leave-One-Out (LOO) diagnostics, we attempt to isolate influential sentences to distinguish between systemic semantic shifts and transient polysemic expansion. Our results show that semantic drift in the fine-tuned Llama-3.1-8B is not evenly distributed across all usages. Instead, a significant part of the change is driven by a smaller set of high-impact contextual instances, rather than gradual and uniform change across all occurrences. This work provides a preliminary framework for low-resource Sinhala diachronic analysis, highlighting the trade-offs between model sensitivity and data availability.

Visual Question Answering (VQA) over charts and tables requires joint reasoning over language, visual structure, and numerical information, yet no standard benchmark currently exists for Vietnamese. We introduce ChartTableViVQA, a Vietnamese VQA dataset with 13,003 question–answer–instruction triplets over 868 chart and table images. Questions cover multiple reasoning types, ranging from simple value lookup to multi-step aggregation, comparison, and trend analysis, and are accompanied by concise instructions that explicitly guide the reasoning process. The dataset is built via a semi-automatic human-in-the-loop pipeline, combining large language model generation with manual verification by Vietnamese annotators. We evaluate several vision-language models in zero-shot and instruction-based settings. Results show strong performance on simple questions but clear weaknesses in complex reasoning, particularly on charts, while instruction-based prompting consistently improves accuracy.

Automatic Speech Recognition (ASR) can ease clinical documentation in resource-constrained regions, but deployment is hindered by a "Reality Gap" between laboratory performance and noisy, real-world clinical telephony, compounded by strict data residency and compute constraints. We study this gap using Gram Vaani, a telephonic Hindi corpus spanning rural healthcare and agricultural helplines, as the closest publicly available proxy for clinical telephony speech, and show that a robust multilingual model (IndicWav2Vec) degrades from 11.60% WER on clean read Hindi to 41.72% WER on this data. We evaluate a progression of adaptation regimes, from full fine-tuning and offline Low-Rank Adaptation (LoRA) upper bounds to an on-device, stream-based continual adaptation framework in which raw audio never leaves the local device, and characterize the trade-offs between data-driven and parameter-driven stabilization strategies. Our evaluation covers both lexical accuracy (WER and CER) and semantic fidelity (BERTScore) on the target domain, alongside the retention of general-domain knowledge. Multi-domain Experience Replay (ER) yields the primary gains, improving target WER by 18.2% relative and reducing catastrophic forgetting by 54% compared to naive adaptation, with BERTScore reflecting consistent gains in semantic fidelity. Combining replay with Elastic Weight Consolidation based on a stabilized importance estimate (Absolute Fisher) yields the strongest retention at a small cost in plasticity. Finally, a language model spot check empirically verifies that the core mismatch lies at the acoustic level and cannot be resolved by language models alone.

Image Captioning lies at the center of synthetic data generation in computer vision. The quality of the caption defines the behavior of the resulting model trained on image-caption pairs generated by Image Captioning. Measuring the specificity of image captioning plays an important role in supplying high-quality captions for the downstream tasks. To address the challenge of automatically assessing the quality of the caption, we propose SPECI, an automatic metric that decomposes dense captions into atomic facts and evaluates specificity across visual criteria such as granularity, spatial detail, color, and linguistic focus using WordNet-based hyponym levels. Empirical results on the benchmark datasets demonstrate significant correlation with human judgments, and its application in image captioning and image reconstruction tasks highlights the practical utility of SPECI. The source code of SPECI is available at https://github.com/sonlam1102/SPECI.

Large language models (LLMs) exhibit both knowledge of language, rendered by, for example, grammatical judgment ability, and abilities to perform tasks through language, such as math-word problem solving. Motivated by neurolinguistic studies of their neural dissociation in human brains, in this study, we investigate the overlap of their circuits in large language models (LLMs). Specifically, we examine the overlap and causal effects between the circuits extracted for linguistic and arithmetic tasks, using controlled materials and circuit discovery methods from the mechanistic interpretability field. Our experiments with five LLMs show that circuit similarity is mostly explained by task domains (linguistic vs. arithmetic), rather than by their input and output vocabularies or by the syntactic similarities between natural-language sentences and arithmetic expressions. This finding suggests that LLMs may process linguistic and arithmetic information using largely separate neural circuits, highlighting an ability-based modular organization of circuits within these models.

Activation steering offers a lightweight mechanism for controlling model behavior, but existing methods depend on sentence-level contrastive data that is unavailable for most of the world’s languages. We show that multilingual sentiment lexicons are a viable substitute across three model families: aggregated multilingual lexicons outperform English-derived steering vectors for mid- and low-resource languages, while monolingual lexicons remain stronger for languages a model has specialized in during pre-training. The steering efficacy is mechanistically dictated by both representational density and pre-training priors. Large-magnitude English vectors overwhelm the compact representations of low-resource languages, causing "linguistic collapse." This fragility is mitigated by the norm-dampening of multilingual aggregation. Architectural biases, including bilingual specialization penalty and latent numerical rigidity, also heavily constrain cross-lingual steering responsiveness. Ultimately, our work establishes lexicons as a practical steering signal and offers a comprehensive mechanistic account of cross-lingual steering dynamics.

Understanding the impact of prompt variations on large language model (LLM) outputs is critical. Despite extensive research, two challenges persist: conflated terminology stemming from loosely defined concepts, and inefficient evaluations reliant on output generation. To address these, we first formalize prompt categorization to clearly define prompt robustness and controllability. We then introduce the prompt vector (PV), a compact internal representation that reformulates these properties at the representation level. Building on PV, we propose PRC, an efficient and unified framework to systematically evaluate LLMs under few-shot prompt variations without requiring costly autoregressive decoding. Experiments across diverse instruction-tuned LLMs demonstrate that PRC tightly correlates with traditional output-based metrics while substantially reducing computational overhead. Overall, PRC offers a principled and highly efficient tool for diagnosing LLM behavior.

Human-AI collaboration on complex planning goals is bottlenecked by how LLM interfaces handle context: users must manually curate and re-surface relevant information across long and unstructured chat histories. Despite advances in long-context prompting and memory-augmented retrieval, this burden remains unresolved: users still have to identify and supply the right context at each decision point, regardless of how much the model can store or surface. We propose JumpStarter, a system that enables LLMs to collaborate with humans on complex goals by dynamically decomposing tasks to help users manage context. We specifically introduce task-structured context curation, a framework that breaks down a user’s goal into a hierarchy of actionable subtasks and scopes context to localized decision points, enabling finer-grained personalization and reuse. The framework is realized through three core mechanisms: context elicitation, selection, and reuse. In a within-subjects user study, plans produced with JumpStarter were rated substantially higher in quality than those produced with ChatGPT. A complementary automatic simulation study shows that JumpStarter consistently outperforms ChatGPT baselines, planning and memory agents, and workflow ablations. These findings show that effective human-AI planning depends not on the volume of context provided, but on attaching the right context to the right subtask at the right time.

Large language models are widely used for personal advice (relationship conflicts, financial decisions, mental-health support, etc.), but they often align their response with the user’s position rather than offering independent judgment, even when users need pushback rather than validation. Mitigating this, whether through training objectives, system-level guardrails, or evaluation design, requires knowing what triggers it. But what in the user’s message triggers this behavior is unknown. Natural advice-seeking messages vary simultaneously in semantic content (what is disclosed), linguistic style (how it is written), and narrative framing (whose side is told), but existing benchmarks cannot attribute model behavior to any single cue. Without this attribution, mitigations cannot be targeted, for example, suppressing warmth may harm users with genuine distress but might not fix the model’s tendency to endorse whichever side of a conflict speaks. In this work, we isolate which of these cues drive which aspect of sycophantic behavior across 787 advice-seeking scenarios and observe three patterns emerge: emotional warmth is content-driven; stance endorsement is framing-driven: endorsement drops by ∼50% when the same conflict is retold from the opposing perspective; and recommendation directness is style-driven. We also find that hedging reduces rather than increases model endorsement; models do not simply validate vulnerable-sounding users.

The exponential growth of scientific literature has made personalized, factually grounded paper discovery increasingly difficult. Existing systems suffer from limited personalization, cold-start fragility, or hallucinated explanations. We introduce HybridScholar, an integrated architecture combining a CrossAttention-gated fusion of collaborative filtering and RAG learning candidate-specific gate weights per candidate with a Chain-of-Verification (CoVe) pipeline for factually grounded explanations. Evaluated on the Scholar Inbox dataset, HybridScholar substantially outperforms (NDCG@10=0.2991) strong baselines including LightGCN and Scholar-Inbox. Cold-start analysis shows the RAG fallback prevents hard failure for sparse users (NDCG@10=0.283), with cold NDCG rising monotonically as interactions grow. The CoVe pipeline reduces hallucination rate by 23.8%, confirmed by a human evaluation study showing CoVe-revised explanations are preferred by experts on factual accuracy (76.7% vs. 36.0%) and by end-users on helpfulness (78.8%, p<0.0001). Overall, HybridScholar improves ranking quality, cold-start robustness, and factual reliability for large-scale scholarly research discovery, demonstrated on 15,881 users across 239,618 papers.

LLMs are known to introduce stylistic changes into generated text, yet how these stylistic shifts affect model decisions on scientific tasks remains underexplored. In this paper, we focus on multimodal claim verification, where the goal is to determine whether a textual claim is grounded in a given piece of evidence. We apply two rewriting strategies: natural rewriting, which simulates how researchers routinely use LLMs to polish academic text, and controlled injection, which inserts a single LLM-associated word to isolate the effect of vocabulary choice. We evaluate 11 open-weight models spanning five VLM families and ranging from 2B to 38B parameters. We find that models are robust to these modifications: most show no significant drop in accuracy, and compared to prior work on review-score manipulation, verification appears far more stable. However, consistent probability shifts do occur. Hedging-oriented conditions produce significant shifts across nearly all models, while boosting conditions show a weaker effect and general polishing conditions (e.g., grammar correction, fluency improvement) have little effect.

Argument summarization seeks to generate concise representations of argumentative content while preserving argumentative meaning, yet the field remains conceptually fragmented. In this survey, we provide a unified task definition of argument summarization that encompasses both argumentative text summarization and key point analysis, and introduce a taxonomy along input structure, output representation, and the role of argumentative structure. Using this framework, we analyze the evolution of the field and systematically review existing datasets, methods, and evaluation practices, revealing misalignment that hinders comparability and progress. We further situate argument summarization within the broader context of generic text summarization, identifying practical limitations and intrinsic challenges. Finally, we outline promising future research directions.

Dense retrievers can blur the distinctions between short, jargon-heavy records that share most of their vocabulary. We study Lex-TM, a training-free layer that interpolates dense cosine similarity with the Jensen–Shannon similarity of topic distributions from an LDA model whose topic–word prior is scaled by a Herfindahl–Hirschman (HHI) exclusivity score. On NTCIR-13 MedWeb, cast as retrieving tweets that share a query’s exact symptom-label set, Lex-TM reaches an English MRR of 0.8151 against 0.7081 for BM25+dense reciprocal rank fusion, a significant paired difference; in Japanese and Chinese the differences from fusion are not significant. With Qwen3-Embedding-0.6B, routing improves on dense retrieval significantly in English and Chinese. Three findings bound these gains. SPLADE++ and BGE-M3’s multi-vector mode match or exceed Lex-TM. In English and Chinese the gain depends on the label strings in the indexed tweets: without them, routing no longer improves on dense retrieval. The HHI prior brings no benefit: on short texts it becomes a rarity weight, and a symmetric prior performs as well across five Gibbs seeds. Outside MedWeb, routing matches dense retrieval on Chinese medical QA and lowers nDCG@10 on ChatDoctor, FiQA and long biomedical abstracts. Code, seeds and a changelog of fixed bugs are released.

We present a new dataset to support AI-assisted and human studies of the peer reviewing process, based on the ARR Open Review Platform. Unlike previous similar data collections, ours, ARR-Dv2, not only contains the reviews but also author-reviewer discussions (aka rebuttal phase) and meta-reviews. We present statistical analyses of the data with respect to the overall score and the correlations among the overall score, confidence, and soundness scores. Additionally, we extend the task of Review Score Prediction also to include rebuttal data and analyze its effect on score prediction. Finally, we introduce a new task, Meta Review Score Prediction, based on a set of up to three reviews rather than a single review. Our initial results in a zero-shot setting indicate that the rebuttal data adds valuable information to the score prediction and enables reliable predictions.

Contrastive decoding methods such as DoLa improve the factuality of Large Language Models (LLMs) by contrasting the output distributions of mature and premature layers. However, DoLa’s dynamic layer selection relies solely on divergences in output vocabulary distributions. In this work, we propose three attention-guided strategies—Attention-JSD, Attention-Entropy-Max, and Attention-Entropy-Min—that leverage structural information carried by internal self-attention mechanisms as a signal for layer selection. Experimental results on TruthfulQA demonstrate that our strategies, particularly Attention-JSD and Attention-Entropy-Min, consistently outperform the original DoLa. We observe significant gains on multi-answer metrics (MC2 and MC3), suggesting that attention distributions can provide a more sensitive signal for resolving factual knowledge than output vocabulary distributions.

Long-term memory in Large Language Model (LLM) agents requires selective persistence: only a subset of interactions should be consolidated beyond the current context window. Existing memory systems rely on heuristic importance rules or similarity-based novelty, which remain external to the model’s internal computation. We propose a geometric principle for memory formation: episodic memory can emerge from compression failure in latent representation space. We approximate the manifold of routine LLM activations using Sparse Autoencoders (SAEs) and define representational surprise as reconstruction error relative to this learned manifold. Deviations from routine structure yield elevated residuals, providing an unsupervised, model-internal signal for memory writing. We demonstrate this principle through ReSuME, a surprise-gated memory mechanism that commits turns to memory only when normalized reconstruction error exceeds a calibrated threshold. In long-horizon multi-turn dialogue settings, representational surprise separates routine, critical, and out-of-distribution states, and achieves a superior performance–memory trade-off against established memory-agent systems (MemGPT, EM-LLM) and unsupervised anomaly detectors, while operating without labels and at substantially lower inference cost than LLM-judge controllers. Covariance-aware normalization further enables robust cross-domain calibration. These results suggest that episodic memory gating in neural agents can be grounded in intrinsic latent geometry rather than externally engineered rules.

Large Language Models (LLMs) are now widely used for tasks such as summarization, question answering, and speech-to-text translation, and increasingly for code generation, with tools such as Cursor and Windsurf analyzing massive code repositories and recommending changes. Big tech companies have acknowledged growing reliance on LLM-generated code. Although this improves developer productivity, it also raises the risk of suboptimal and insecure code. We introduce a retrieval framework for code generation that connects geometry-aware prototype sampling with syntax-grounded interpretability analysis. By conditioning AST-based explanations on retrieved demonstrations, our method evaluates retrieval quality through both functional correctness and interpretability alignment. In our experiments, high-quality in-context learning (ICL) demonstrations make outputs easier to interpret and improve P@10, whereas poorly chosen demonstrations reduce P@10 relative to the base model. Overall, our approach highlights the importance of efficient sampling strategies for ICL, which can affect model performance on any given task.

Large Language Models (LLMs) have made substantial progress in multilingual question answering, yet their ability to understand low-resource sociolinguistic contexts remains underexplored. We study North Korea as a unique testbed where decades of division have produced distinct differences from South Korea in vocabulary, orthography, norms, and ideological discourse. We introduce LENS (Language and sociocultural Evaluation for North Korean Society), a human-validated benchmark of 1,765 questions evaluating LLMs across three dimensions: linguistic robustness, sociocultural perspective alignment, and domain-specific knowledge. Evaluating diverse recent LLMs, we find a modest performance drop on North Korean language inputs, a larger gap in sociocultural perspective alignment, and severe limitations on domain-specific knowledge. Qualitative analysis reveals that models often conflate formal institutions with lived realities, project liberal-democratic assumptions, or rely on coarse stereotypes. Our results demonstrate that linguistic similarity alone does not guarantee sociocultural or domain-specific understanding, highlighting the critical need for grounded evaluation resources for underrepresented cultural and political contexts.

Large language models (LLMs) are increasingly deployed across high-impact domains, from clinical decision support and legal analysis to hiring and education, making fairness and bias evaluation before deployment critical. However, existing evaluations lack grounding in real-world scenarios and do not account for differences in harm severity, e.g., a biased decision in surgery should not be weighed the same as a stylistic bias in text summarization. To address this gap, we introduce HALF (Harm-Aware LLM Fairness), a deployment-aligned framework that assesses model bias in realistic applications and weighs the outcomes by harm severity. HALF organizes evaluation datasets into harm tiers (Severe, Moderate, Mild) based on the specific task and bias type they measure. Harm severity is assigned according to deployment-relevant criteria grounded in regulatory and risk assessment frameworks. Our evaluation results across eight LLMs show that (1) LLMs are not consistently fair across datasets, (2) model size or performance do not guarantee fairness, and (3) reasoning models perform better in medical decision support but worse in education. We conclude that HALF exposes a clear gap between previous benchmarking success and model fairness. All data and code are available at https://github.com/mbzuai-nlp/HALF

Understanding the multilingual mechanisms of large language models (LLMs) provides insight into how they process different languages, yet this remains challenging. Existing studies often focus on individual neurons, but their polysemantic nature makes it difficult to isolate language-specific units from cross-lingual representations. To address this, we explore sparse autoencoders (SAEs) for their ability to learn monosemantic features that represent concrete and abstract concepts across languages in LLMs. While some of these features are language-independent, the presence of language-specific features remains underexplored. In this work, we introduce SAE-LAPE, a method based on feature activation probability, to identify language-specific features within the feed-forward network. We find that many such features predominantly appear in the middle to late layers of the model and are interpretable. These features influence the model’s multilingual performance and language output, and can be used for language identification with performance comparable to fastText, along with more interpretability. Our code and complete figures are available [here](https://github.com/LyzanderAndrylie/language-specific-features).

Current LLM-based question answering systems provide cited evidence for their claims. However, determining whether each claim within the generated responses is fully supported by its cited evidence is still an ongoing research problem. The task of evaluating attribution quality, or faithfulness to the source text, goes beyond the traditional idea of lexical matching used for relevance in early QA tasks, and the use of lexical models based on semantic distance or word-overlap has declined in favor of NLI-based models and large fine-tuned models. The larger fine-tuned models tend to perform best on current special-purpose benchmarks such as AttributionBench despite struggling with OOD data. Still, exactly why human judgments of faithfulness should align better with LLM evaluators than those based on lexical entailment has never been fully explained. We address the problem with current entailment models by introducing bidirectionality and codifying that idea into a new evaluation modeling approach. We show that this approach is both lightweight and extensible yet competitive with SOTA LLM-based methods, avoiding the pitfalls of both standard NLI and word-overlap approaches. We further show that our model architecture is motivated by a deeper understanding of the gap between lexical distance and faithful claim attribution, exposing the conceptual mechanism behind attribution quality evaluation.

Large language models are increasingly used to annotate texts, but their outputs reflect some human perspectives better than others. Existing methods for correcting LLM annotation error assume a single ground truth. However, this assumption fails in subjective tasks where disagreement across demographic groups is meaningful. Here we introduce Perspective-Driven Inference, a method that treats the distribution of annotations across groups as the quantity of interest, and estimates it using a small human annotation budget. We contribute an adaptive sampling strategy that concentrates human annotation effort on groups where LLM proxies are least accurate. We evaluate on politeness and offensiveness rating tasks, showing targeted improvements for harder-to-model demographic groups relative to uniform sampling baselines, while maintaining coverage.

Large Language Model (LLM)-based UI agents show great promise for UI automation but of- ten hallucinate in long-horizon tasks due to their lack of understanding of the global UI transition structure. To address this, we in- troduce AGENT+P, a novel framework that leverages symbolic planning to guide LLM- based UI agents. Specifically, we model an app’s UI transition structure as a UI Transi- tion Graph (UTG), which allows us to refor- mulate the UI automation task as a pathfinding problem on the UTG. This further enables an off-the-shelf symbolic planner to generate a provably correct and optimal high-level plan, preventing the agent from redundant explo- ration and guiding the agent to achieve the au- tomation goals. AGENT+P is designed as a plug-and-play framework to enhance existing UI agents. Evaluation on the AndroidWorld benchmark demonstrates that AGENT+P im- proves the success rates of state-of-the-art UI agents by up to 14.32% and reduces the ac- tion steps by 37.70%. Our code is available at: https://github.com/KiteFlyKid/agentp.

Large language models are now deployed as annotators across NLP, generating training labels and evaluation scores, yet the field has no established standard for verifying whether LLM-generated labels are reliable, internally consistent, or comparable across studies. This methodological gap does not begin with LLMs: a systematic corpus study of 665 ACL Anthology papers from 1999 to 2026 reveals that even classical human annotation in NLP suffers from systematic measurement and reporting failures that have left a growing body of work scientifically incomparable. Papers were identified through keyword search and organised via topic modelling into a thematic taxonomy spanning eight annotation domains from named entity recognition to disagreement modelling. Full-text analysis of 549 papers yields three principal findings. First, Cohen’s kappa dominates across all task types (259 papers, 38.9%), regardless of whether its assumptions are satisfied; ICC, appropriate for continuous-scale tasks such as MT evaluation, appears in only 16 full-text papers (2.4%). Second, 226 of the 665 papers (34.0%) name no inter-annotator agreement (IAA) coefficient anywhere in the full text, which is a methodology-reporting concern; a further 50.1% of papers name a coefficient in the body but not in the title or abstract, so 84.1% of papers omit the metric from the abstract layer. We frame the body-only gap as a discoverability concern (the methodology is reported but is not visible at the abstract layer readers and reviewers encounter first), distinct from the full-text omission. Third, annotator disagreement modelling and LLM-as-annotator constitute the fastest-growing area of the corpus (124 papers, 18.6%): 26 of these appeared between 1999 and 2022 and 98 between 2023 and 2026, the only dimension whose volume rises in that window. Fewer than 5% of papers report confidence intervals. We provide a large-scale empirical analysis of metric reporting practice within the ACL Anthology and a seven-point agenda for standardised reporting, complementing prior normative guidance with field-scale descriptive evidence.

This paper presents a case study on the development of Auto-AdvER, a specialised span-labelling schema, along with its corresponding dataset for the textual car advertisement domain. Developed with industry needs in mind, Auto-AdvER is designed to enhance text mining analytics in this domain and contributes a linguistically unique sequence-labelling dataset. We introduce a schema consisting of three labels: ’Condition’, ’Historic’, and ’Sales Options’. We outline the guiding principles for annotation, describe the methodology for schema development, and present the results of an annotation study demonstrating an inter-annotator agreement of 78%. Furthermore, we compare the performance of various models, including transformer encoder models (BERT, DeBERTaV3) and transformer decoder-based open- and closed-source large language models: Llama, Qwen, GPT, and Gemini, as well as conditional random fields. Our results show that a fine-tuned DeBERTaV3 encoder achieves the highest F1-score, followed closely by conditional random fields, while the large language models reach the highest precision but lower recall. We present this work as a stepping stone towards more fine-grained analysis and discuss Auto-AdvER’s potential impact on advertisement analytics, pricing, and customer insights. Potential applications include the analysis of market dynamics and data-driven predictive maintenance.

Vision-Language Models (VLMs) enable video annotation at scale, but costs accumulate quickly: processing a typical 60-second short-form video at one frame per second requires millions of tokens. To reduce costs, researchers rely on heuristics such as sampling a subset of frames, compressing videos into image grids, or using only a single modality. However, it remains unclear which heuristics save cost, and whether they preserve the downstream conclusions these annotations enable. To address this gap, we conduct a systematic evaluation of these heuristics using short-form videos, on two computational social science (CSS) tasks: sentiment and topic classification. We evaluate each configuration along three axes the literature typically treats separately: classification accuracy, validity of downstream inference, and per-video token cost. First, we find that accuracy and validity diverge: the highest-accuracy configuration can produce wrong conclusions. Second, modality value is not guaranteed: text alone can yield strong performance, indicating that adding modalities can add cost without adding signal. Finally, we find that cost can be decoupled from video length when annotating short-form videos: a single 2x8 image grid built via simple shot-transition detection approaches full-video understanding (kappa within .05), at ~15% of the token cost. Based on these findings, we derive guidelines that can enable cost-aware VLM annotation in CSS.

Recent advancements in large language models have introduced new capabilities for reasoning over structured data, particularly through program-aided tools that can analyze tables. However, many existing methods address single-table scenarios or assume that the relevant tables are already provided. In practice, users often issue complex data exploration queries over entire databases, where relevant information may be distributed across multiple relations. In this work, we introduce DBRAG, a retrieval-augmented generation framework tailored for multi-table question answering. DBRAG first retrieves candidate tables using an offline table index, enriches their summaries with query-relevant rows, and uses an LLM to rerank the candidates. A program-aided reasoner then selects the required tables and executes operations over their full contents, keeping the initial prompt context compact. Experiments on the Spider, GeoQuery, and ATIS datasets used in this study demonstrate improvements in table retrieval and multi-table question answering.

Multi-Agent (MA) systems are effective at solving complex tasks that demand planning, tool use, and the synthesis of evidence from multiple sources. Existing systems typically adopt Hierarchical Manager-Worker (HMW) or Router-based Message Passing (RMP) structures as their communication protocol. However, these designs restrict agent autonomy: Worker agents cannot directly consult specific “peers”, and misrouted messages can propagate errors. Inspired by bus architectures in computer systems, we propose BusMA, a communication framework that allows any agent to address other agents through a shared channel, i.e., the Bus. It consists of agent registration, message routing, and shared memory management components. Worker agents, each equipped with tools, have their own local memory and can reason, act (tool usage), and communicate by posting shared messages with specific intents. We introduce four intents: discussion, challenge, guidance, and request for explanation, which support fine-grained communication among agents. A Chair agent monitors the shared memory to coordinate interactions and facilitate convergence among Workers. To evaluate the effectiveness of BusMA, we conduct extensive experiments with two frontier LLMs across 13 tasks spanning visual reasoning, mathematical reasoning, and knowledge retrieval. The results demonstrate that BusMA consistently outperforms state-of-the-art HMW and RMP methods.

In visually-rich documents, information is encoded not only in individual page objects such as tables, headers, and text blocks, but also in the structural relations among them, making document structure analysis fundamental to information retrieval and document understanding. However, accurately inferring such relations remains challenging in multi-page documents with long-range dependencies and heterogeneous layouts. To address this, we propose a tree-guided and self-regressive framework, termed DOcument Structure Analyzer (DOSA), for inferring relations among page objects and reconstructing document-level semantic trees. DOSA processes documents chunk-by-chunk, fusing visual, textual, and layout features for each page object and predicting hierarchical and ordering relations. The predicted relations are used to incrementally construct a semantic tree, which is then leveraged as structural context to guide inference on subsequent chunks. Experimental results on five benchmarks demonstrate the effectiveness of DOSA, with improvements of up to 4 F1 points and 19 TEDS points on DocHieNet, the most challenging multi-page hierarchy benchmark.

Discrete diffusion language models can generate text efficiently by updating multiple masked positions in parallel, but this parallelism introduces a quality-latency trade-off. Aggressive decoding may commit mutually dependent tokens too early, while conservative decoding requires many denoising steps. Existing methods address this tension by deciding which tokens are safe to reveal using confidence or dependency criteria. However, avoiding unsafe commits does not necessarily make the remaining masked sequence easy to decode, since uncertain tokens may depend on masked tokens, creating a bottleneck for denoising steps. We propose AXON, a training-free module that can be added on top of existing parallel decoding strategies for diffusion language models. Rather than replacing the base decoder, AXON monitors the remaining uncertain masked tokens and intervenes only when their current state suggests that additional context is needed. It then shifts the criterion from which tokens are safest to reveal to which confident reveals would best support later denoising. AXON selects anchors, confident masked tokens that uncertain positions attend to, using attention, uncertainty, and confidence signals. Experiments on reasoning and code-generation benchmarks across multiple diffusion language models show that AXON improves the quality-latency trade-off of existing parallel decoders, often reducing the number of function evaluations while maintaining or improving accuracy.

Black-box distillation trains student models using only teacher-generated outputs, making it practical when teacher models are accessible only through APIs. However, existing methods either rely mainly on positive imitation or require costly full-response rollouts to obtain corrective signals. We show that such corrective signals can be obtained locally from teacher trajectories alone. At each position, the teacher token provides a positive anchor, while a high-probability alternative sampled from the student provides an informative negative. Based on this insight, we propose Token-level On-policy contrastive Learning for blackbox Distillation (TOLD), which samples a single negative token from the student’s distribution and optimizes a Bradley-Terry objective anchored by negative log-likelihood. This enables the student to reinforce teacher tokens while suppressing its own likely mistakes, without teacher logits, reward models, or full-response rollouts. Experiments on reasoning and instruction-following tasks show that TOLD outperforms or matches strong black-box distillation baselines and achieves up to 1.88 times training speedup over rollout-based on-policy baselines.

Autonomous agents based on large language models (LLMs) are becoming an increasingly prevalent paradigm for tackling complex and real-world tasks. Despite the remarkable zero-shot capabilities of modern LLMs, specialized agent training is often essential to obtain reliable and improved performance for specific target tasks. In this survey, we provide a process-centric review of the training of LLM-based agents, with a primary focus on methods that update the parameters of the main agent policy. Rather than organizing prior work only by application domain, agent capability, or learning algorithm, we examine the end-to-end pipeline through which LLM-based agents are trained: environment setup, data and trajectory construction, learning-signal design, and optimization objectives and schemes. This perspective highlights how design choices at different stages interact with one another. Finally, we conclude with a discussion on potential future directions.

Despite advances in safety alignment, prompt-rewriting attacks such as persona modulation, fictional framing and persuasion-based reformulation, can bypass safety filters even on frontier models. Existing defenses either rely on non-scalable human curation or white-box optimisation that overfits to specific model internals, leaving aligned models brittle against the very class of adaptive black-box adversaries they will face in deployment. To address this gap, we introduce CHASE (Co-evolutionary Hardening through Adversarial Safety-Escalation), a closed-loop red-blue teaming framework in which a black-box attacker and a safety-aligned defender co-evolve. The attacker is trained via Group Relative Policy Optimization (GRPO) under a multiplicative reward that jointly enforces bypass effectiveness and intent fidelity, while the defender is hardened on the harvested adversarial rewrites through a two-stage GRPO + rejection-sampled SFT pipeline balanced with benign data. Evaluated on BeaverTails and JailbreakBench against five held-out attack families (PAIR, TAP, AutoDAN, PAP, Translation), CHASE cuts mean StrongREJECT score by 43.2% on Llama-3.1-8B-Instruct with 0% false refusal on benign (Alpaca) prompts, at a measured helpfulness cost of a 55.6% refusal rate on XSTest safe prompts and a 1.92-point MT-Bench reduction. A controlled ablation that varies only the training attack distribution shows that this generalisation comes from the template-free attack distribution rather than the pipeline, and the result replicates on Qwen2.5-7B-Instruct. Across the two similarly sized open models we study, these results suggest that template-free RL exploration recovers latent attack primitives that transfer across mechanistically distinct attack families.

Personalized text generation requires models not only to produce coherent text but also to align with a target user’s style, tone, and topical focus. Existing retrieval-augmented approaches such as LaMP and PGraphRAG enrich profiles with user and neighbor histories, but they stop at generation and often yield outputs that drift in tone, topic, or style. We present PerFine, a unified, training-free critique–refine framework that enhances personalization through iterative, profile-grounded feedback. In each iteration, an LLM generator produces a draft conditioned on the retrieved profile, and a critic LLM - also conditioned on the same profile - provides structured feedback on tone, vocabulary, sentence structure, and topicality. The generator then revises, while a novel knockout strategy retains the stronger draft across iterations. We further study additional inference-time strategies such as Best-of-N and Topic Extraction to balance quality and efficiency. Across Yelp, Goodreads, and Amazon datasets, PerFine consistently improves personalization over PGraphRAG, with G-Eval gains of +8.5–12%, steady improvements over 3–5 refinement iterations, and scalability with increasing critic size. These results highlight that post-hoc, profile-aware feedback offers a powerful paradigm for personalized LLM generation that is both training-free and model-agnostic.

Recent advances in Large Language Models (LLMs) have expanded their role in human interaction, yet information-seeking behavior, particularly across cultural contexts, remains underexplored in these systems. In this work, we study cross-cultural variation in information-seeking using question-asking as a mechanism. We introduce IQUEST (Information-seeking QUestion Evaluation across SocieTies), an evaluation framework that measures human-model alignment in information-seeking question behavior through linguistic (style), topic preference (content) analysis and grounding insights in social science constructs. Across open- and closed-source models, we find that LLMs flatten cross-cultural diversity, aligning more closely with question-asking patterns expressed in Western countries. We also explore fine-tuning strategies, which narrow the human-model alignment gap by 43% overall. Finally, we demonstrate that stronger information-seeking question behavior improves downstream cultural adaptability, highlighting its importance for inclusive NLP research.

Efficient context compression is critical for retrieval-augmented question answering in resource-constrained settings, where long retrieved contexts increase latency, memory use, and LLM reader cost. We propose a lightweight encoder-only framework for query-driven sentence pruning that preserves answer-critical evidence while aggressively reducing irrelevant context. Our method learns marginal contribution scores for sentences using counterfactual training signals and optimizes a contrastive ranking objective that separates critical evidence from noncritical context. Our approach scores all sentences from a single full-context encoding, enabling fast inference with low computational overhead. Experiments show that it maintains accuracy comparable to the strongest baseline while using 3.7× less peak memory and achieving nearly 3× lower compression latency, demonstrating an

Abstention is the intentional refusal of models to provide an answer. Adverse drug reactions (ADRs), which range from mild to life-threatening, are often underreported by patients in hospitals and to doctors because patients assume these are unavoidable. With the growing adoption of LLMs and vision language models (VLMs) in healthcare, patients are increasingly reaching out to these models for solutions. In such high-risk scenarios, incorrect or overconfident responses can cause unnecessary anxiety, lead patients to discontinue essential medications, or encourage inappropriate self-medication. In this work, we propose the MAD-MED, a Model Abstention approach for Drug Adverse Reaction for MEDical-VLMs. It is a training strategy to encourage models to abstain when answering would contradict the available input, exceed the model’s knowledge, or violate model ethics. In addition, we release an augmented Adverse Drug Reaction ABStention (ADR-ABS) dataset comprising image-answerable, text-answerable, and unanswerable questions to support systematic evaluation of abstention behavior. Quantitative and human evaluations confirm that our proposed MAD-MED approach improved the abstention capabilities of models considerably while keeping the generative capabilities intact in cases where the model is expected to answer.

Chinese text correction has traditionally focused on spelling and grammar, while factual error correction is usually treated separately. In paragraph-level Chinese professional writing, linguistic (word/grammar/punctuation) and factual errors can co-occur and interact. Moreover, editorial review leaves few observable errors in published professional texts, making controlled benchmark construction important. This paper introduces **CLFEC (Chinese Linguistic & Factual Error Correction)**, a new task for joint linguistic and factual correction. We construct a mixed, multi-domain Chinese professional writing dataset spanning current affairs, finance, law, and medicine. We then conduct a systematic study of LLM-based correction paradigms, from prompting to retrieval-augmented generation (RAG) and agentic workflows. The analysis reveals practical challenges, including limited transfer of a specialized corrector, the need for evidence grounding for factual repair, the difficulty of mixed-error paragraphs, and over-correction on clean inputs. Results further show that handling linguistic and factual errors within the same context outperforms decoupled pipelines, and that agentic workflows can be effective with suitable backbone models. Overall, CLFEC provides a new benchmark for Chinese text correction research and practical guidance for proofreading systems. The dataset is publicly available at [https://github.com/jiu2021/CLFEC-Dataset](https://github.com/jiu2021/CLFEC-Dataset).

Legal RAG systems often retrieve passages by semantic relevance, lexical score, or citation match. In versioned legal corpora, however, a passage can be topically relevant and citation-matched while being legally invalid for the operative date. We study law-in-force retrieval: retrieving evidence that matches the authority, is valid at the operative date, and supports the answer. We build LawInForce-eCFR from 805 human-validated eCFR/CFR amendment pairs and 3,220 controlled answer-flipping yes/no questions. We also present AuthorityFirst-RAG, a transparent diagnostic retriever that decomposes retrieval into authority identification, temporal scoping, and rule-support ranking. AuthorityFirst-RAG improves Recall@1 from 2.80% for BM25-date-filter to 30.90%, but generation remains difficult even with GoldOracle evidence. These results show a dual bottleneck in versioned legal RAG: retrieving the law in force and applying numeric legal rules.

Large Language Models (LLMs) are fluent but prone to hallucinations, producing answers that appear plausible yet are unsupported by available evidence. This failure is especially problematic in high-stakes domains where decisions must be justified by verifiable information. We introduce EvidenceRL, a reinforcement learning framework that enforces evidence adherence during training. EvidenceRL scores candidate responses for grounding (entailment with retrieved evidence and context) and correctness (agreement with reference answers) and optimizes the generator using Group Relative Policy Optimization (GRPO). We evaluate across two high-stakes domains, cardiac diagnosis and legal reasoning, where EvidenceRL consistently improves evidence grounding and faithfulness without sacrificing task accuracy. On cardiac diagnosis, F1@3 increases from 37.0 to 54.5 on Llama-3.2-3B while grounding (G_max@3) rises from 47.6 to 78.2; hallucinations drop nearly 5x and evidence-supported diagnoses increase from 31.8% to 61.6%. On legal reasoning, EvidenceRL raises Faithfulness from 32.8% to 67.6% on Llama-3.1-8B, demonstrating consistent behavioral change across domains.

While individual components of agentic architectures have been studied in isolation, we lack empirical evidence on how design dimensions interact in realistic multi-agent systems. This study aims to address these gaps by providing a comprehensive enterprise-specific benchmark. We introduce AgentArch: a benchmark that evaluates 18 agentic configurations across state-of-the-art large language models. The study spans four dimensions: orchestration strategy, agent style (ReAct versus function calling), memory architecture, and thinking tool integration. Results reveal strong model-specific architectural preferences that challenge one-size-fits-all assumptions in agentic AI. They also expose notable performance gaps on enterprise tasks: the best models reach only 35.3% success on the complex task and 70.8% on the simpler task. We aim to inform architecting choices for enterprise agents by grounding model selection and component design in systematic evidence.

Large Language Models utilizing reasoning techniques improve task performance but incur significant latency and token costs due to verbose generation. Existing automatic prompt optimization(APO) frameworks target task accuracy exclusively at the expense of generating long reasoning traces. We propose Cost-Regularized Optimization of Prompts (CROP), an APO method that introduces regularization on response length by generating textual feedback in addition to standard accuracy feedback. This forces the optimization process to produce prompts that elicit concise responses containing only critical information and reasoning. We evaluate our approach on complex reasoning datasets, specifically GSM8K, LogiQA and BIG-Bench Hard. We achieved an average 68% reduction in token consumption across diverse reasoning benchmarks (and up to 80.6% on algorithmic tasks) while maintaining competitive accuracy, seeing only a nominal decline in performance. This presents a pragmatic solution for deploying token-efficient and cost-effective agentic AI systems in production pipelines.

LLM-as-a-Judge frameworks offer scalable alternatives to human evaluation but inherit the limitations of text-only reasoning, particularly in tasks requiring precise computations. We introduce J-ToRL, a framework that trains LLM judges to leverage computational tools via code generation through end-to-end reinforcement learning (RL). Unlike prior approaches relying on inference-time integration, J-ToRL internalizes tool-use strategies as part of its policy, enabling exploration of diverse reasoning pathways. We identify a key challenge in applying RL with verifiable rewards to judgment: the coarse reward signals inherent in multiple-choice verdicts can incentivize shortcut learning. J-ToRL mitigates this via an auxiliary verifiable reward mechanism based on reference-answers, that enforces stricter criteria beyond simple verdict accuracy alone. With a sophisticated pairwise comparison dataset across five domains, J-ToRL-trained LLMs improve the base models, outperform prior judges, and close the gap to frontier LLMs. Further, J-ToRL-trained models exhibit beneficial cognitive behaviors such as self-error-recovery and self-refinement of tool use strategies.

Personalized multi-document summarization (MDS) is essential for meeting individual user preferences of writing style and content focus for summaries. In this paper, we propose that for effective personalization, it is important to identify fine-grained differences between users’ preferences by comparing preferences of different users. Motivated by this, we propose ComPSum, a personalized MDS framework. It first generates a structured analysis of a user by comparison between users. The generated structured analysis is then used to guide the generation of personalized summaries. To evaluate ComPSum, we propose AuthorMap, a fine-grained reference-free evaluation framework for personalized MDS. It evaluates the personalization of a system based on the authorship attribution between two personalized summaries generated for different users. For robust evaluation of personalized MDS, we construct PerMSum, a personalized MDS dataset in the review and news domain. We evaluate the performance of ComPSum, showing that it outperforms strong baselines. The code is available at https://github.com/leehaoyuan/comparative_personalize_sum.

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced reasoning capabilities in Large Language Models. However, adapting RLVR to multimodal domains suffers from a critical perception-reasoning decoupling. Existing paradigms, driven by text-centric outcome rewards, reasoning in language medium, inadvertently encourage models to bypass visual perception. We empirically validate this through blind experiments: paradoxically, state-of-the-art policies maintain or even improve performance even when visual inputs are entirely removed. This reveals that these models degenerate into blind reasoners, exploiting linguistic priors to generate plausible answers instead of attending to visual evidence. In response, we propose Thinking with Deltas, a framework driven by a Differential Visual Reasoning Policy (DVRP). DVRP introduces intrinsic supervision via visual triplets, comprising original, masked, and perturbed inputs. It optimizes the model to maximize reasoning divergence from masked inputs (enforcing visual sensitivity) while minimizing divergence from perturbed inputs (ensuring visual robustness). By aligning reasoning variations strictly with the Delta of visual information, DVRP inherently bolsters visual understanding capabilities and significantly outperforms state-of-the-art methods on both general and medical benchmarks, without requiring external annotations or auxiliary tools.

When an LLM judge scores an output, its score distribution retains uncertainty and disagreement information that is lost after scalar compression. We introduce JudgeMoE, a lightweight aggregator that assigns example-specific weights to cached judge score distributions and fuses them before computing a final score. A protocol study shows that score-range choice is unstable across judge–dataset settings and that soft scoring usually outperforms hard decoding. On the original 10-cell benchmark, JudgeMoE improves mean Spearman over uniform log pooling by +0.079. Applying the same configuration to six additional cells yields a +0.0393 mean gain over the strongest local single judge across 16 cells, with positive differences in 12/16 cells and a one-sided Wilcoxon signed-rank p=0.0091. Validation-based analyses further show that the preferred aggregation method depends on the task and judge pool.

Jailbreak attacks are usually compared by their attack success rate (ASR), with a higher ASR read as a stronger attack. But an ASR mixes an attack’s general strength with its fit to the particular model it was measured on, and it also depends on the judge that scored it. We separate these factors with a matched evaluation grid: five attacks on eight open-weight vision–language models, 200 HarmBench behaviours per cell, every turn scored by three judges (Qwen2.5-7B, Qwen2.5-72B and Llama-Guard-3), and each ASR decomposed into general strength, target susceptibility and target-specific fit. Attack rankings invert across targets: FigStep leads PAIR on InternVL2.5-8B (13.0% vs. 3.5%, paired p=4×10^-10) and trails it on Qwen2.5-VL-7B, in the same direction under all three judges. Across the grid, the target-specific component is statistically significant under every judge (p<10^-12). The judge matters as well: agreement between judges is low (Cohen’s 𝜅 between 0.03 and 0.29), and a larger judge with the same rubric changes which attack ranks first on three of eight targets. A higher ASR therefore shows that an attack succeeded more often on a given model under a given judge, but not whether its advantage is general or specific to that model. We release the protocol and ∼20,400 trajectories with per-turn scores under all three judges

Negotiation is a core component of social intelligence, requiring agents to balance strategic reasoning, cooperation, and social norms. Recent work shows that LLMs can engage in multi-turn negotiation, yet nearly all evaluations occur exclusively in English. We systematically isolate language effects across English and three Indic framings (Hindi, Punjabi, Gujarati) by holding game rules, model parameters, and incentives constant for multi-agent simulations for Ultimatum, Buy-Sell, and Resource Exchange games. Our results indicate that language choice is correlated to agreement and process dynamics, while final payoffs are statistically unchanged, and that this correlation persists across model architectures. Crucially, effects are task-contingent: Indic languages lower acceptance and longer negotiations in the Ultimatum Game, and higher trade volume in Resource Exchange. Further, they suggest that findings obtained under English-only conditions may not generalize to other linguistic settings. Even within one related language family, we observe task-contingent divergence from English, which cautions against treating English-only evaluation as representative. Whether this extends to typologically distant languages remains open.

Agentic Multimodal Large Language Models (MLLMs) are reshaping scientific discovery. They do more than answer questions: they read multimodal evidence, reason over scientific constraints, use tools, and close parts of the research loop. This survey explains how such systems are built for scientific discovery, moving from multimodal perception to training, inference, evaluation, and human-AI collaboration. Across this pipeline, scientific agents face bottlenecks that generic assistants often hide, including scarce expert annotations, evolving domain structures, complex reasoning, immature tool interfaces, safety and toxicity risks, and unfaithful results. We highlight design choices for turning agentic MLLMs into trustworthy, evidence-grounded collaborators. Curated resources are available at https://github.com/JianJinglin/awesome-agentic-AIScientists.

Conversational Intelligent Tutoring Systems (CITS) represent a convergence of artificial intelligence, learning sciences, and human-computer interaction, aiming to provide the adaptive and dialogic learning as expert teachers. In the past decades, the advancement of machine learning and natural language processing reshaped CITS’s design, implementation, and evaluation. In this survey, we analyze the evolution of computational paradigms in shifting pedagogical affordances, interaction styles, and methodological development. We frame CITS not merely as dialogue systems, but as proactive agents that reason about learner cognition, affect, and context while enacting pedagogical strategies grounded in educational theories. From domains, populations, and instructional goals, we identify key components, evaluation approaches, and unresolved challenges. Our paper highlights the promise and limitations of conversational tutoring in the generative AI era, offering a foundation for future research that seeks to maximize their pedagogical value.

Instruction-following machine translation (IF-MT) requires respecting prompt-level rules on terminology, formatting, and register. Rule compliance typically trades off against translation quality, a tension that general-purpose IF data augmentation methods do not address. We propose Reference-Grounded Data Curation, a two-phase pipeline that extracts every supervised constraint from a reference translation that already satisfies it, ensuring feasibility by construction. Phase 1 applies Instruction-Following Difficulty (IFD) scoring to retain the hardest-but-learnable instances from an English-Thai parallel pool. Phase 2 extracts constraints from each reference target and keeps only generations satisfying every constraint, yielding the 1.97M-record Grounded dataset. We fine-tune open-weight bases on Grounded to produce ChindaMT, a Thai-English translation family at 4B, 2B, and 0.8B parameters. Under length-controlled pairwise judging, ChindaMT outperforms or matches every same-size baseline at every tier on both plain translation and under explicit rules, reaching up to a 68.4% win rate against the strongest baseline. The recipe transfers cleanly across Qwen generations. We release model weights, the Grounded dataset, and evaluation suites.

Large language model (LLM) evaluation usually averages scores across many benchmark tasks, implicitly assuming broad task coverage is necessary for reliable general evaluation. This practice also implicitly treats all benchmark tasks as equally important for general evaluation. We revisit this assumption by treating benchmark results as a model-task score matrix under the performance prediction task. We find that a broad range of benchmarks are redundant with scores for 100 LLMs on 13 aggregate tasks and 523 subtasks. Specifically, we recovered model rankings with high rank correlation with the ALS matrix completion method after masking out several task scores. This means that no tasks dominated the recovery process, and both low-scoring and high-scoring tasks shared the same trends. Further analyses show that multiple-choice scores allow us to recover model rankings on free-form generation tasks. For general model ranking, free-form generation evaluation is not necessarily more informative than multiple-choice evaluation. These findings suggest that evaluating general-purpose LLMs can be accomplished with fewer tasks than the current common practice. Our analysis also estimates the number of models required to capture the overall trends in the benchmarks.

Despite the impressive reported performance of Large Language Models (LLMs) on medical Question-Answering (QA), evaluation still overrelies on multiple-choice and extractive settings primarily assessing textbook knowledge, with limited work on open-ended QA over patient records. We present a comprehensive study investigating LLM long-context (LC) comprehension on longitudinal, patient-centric Electronic Health Records (EHRs) across multiple datasets, models and metrics, covering extractive, multiple-choice, and open-ended generative QA under different context formulation and Retrieval Augmented Generation (RAG) settings. We show that medically fine-tuned models offer no measurable benefit over base counterparts, smaller models exhibit stronger memorization, and longitudinal supplementary notes aid generative QA. Reasoning models gain on complex multi-note QA under LLM Correctness but not stricter reference-aligned metrics, while hybrid RAG with reranking outperforms Full-Context (FC) prompting. A question-category analysis and multi-faceted metric evaluation further reveal where RAG and FC excel and struggle, with RAG falling behind in longitudinal reasoning, thus uncovering systematic failures pointing to the need for reasoning-appropriate metrics.

Video-guided machine translation (VMT) translates text by leveraging the context provided by the accompanying video. This additional contextual information helps improve translation quality by disambiguating ambiguous words or explicitly indicating attributes such as the colour or gender of objects and people. Prior works have extensively explored VMT; however, most approaches train models from scratch, which requires large amounts of data. Additionally, the majority of VMT research has focused on the English–Chinese language pair. In this paper, we explore training strategies for VMT using only a limited amount of data and extend the task to Indic languages, specifically Hindi, Marathi and Odia. We also propose a synthetic data framework that is effective for training VMT models in low-resource language settings. Our results demonstrate an improvement of +1.75 BLEU compared to baseline models.

Large language models (LLMs) perform strongly in machine translation, but their effectiveness as translation refiners depends on the draft source. We compare direct translation, self-refinement, and cross-system refinement on English-to-Chinese WMT test sets. Sockeye drafts obtain higher lexical-overlap scores but lower COMET than the stronger Qwen3 drafts, and receive larger gains after Qwen3 refinement. Additional controls qualify this pattern. Qwen3-0.6B drafts obtain the largest raw gain but lower final quality, while Qwen3-1.7B drafts start near Sockeye in COMET yet benefit less. Cross-family GLM-4-9B-Chat drafts also improve with Qwen3-8B, but achieve smaller gains and lower final quality than refined Sockeye drafts. Descriptive bucket, common-support, and regression analyses suggest that initial quality alone is an incomplete account of these differences. xCOMET-XL score and span diagnostics further characterize the contrast with Qwen3-8B self-refinement. We frame translation editability as a property of the evaluated draft–editor pairing, assessed jointly through quality gain, improvement probability, degradation risk, and final quality.

For very low-resource language pairs such as Bahnar-Vietnamese, the relative effectiveness of generic multilingual, language-pair-specific lexical, and task-adapted retrieval remains unclear. We study Bahnar-Vietnamese parallel-sentence retrieval (bitext retrieval) as a controlled candidate-ranking task and compare lexical, alignment-based, neural, and hybrid methods under a development-based selection protocol. The strongest off-the-shelf encoder, LaBSE, reaches 24.24% Recall@1, while symmetric IBM1 reaches 49.33%. Among four in-domain modern dense retrievers, development selection chooses DPR-XM, which reaches 47.43% Recall@1 and 72.21% Recall@10. XLM-R LoRA reaches 64.02% Recall@10, and the development-selected XLM-R-first IBM1 reranker gives the highest Recall@1 point estimate, at 51.62%, with an MRR of 0.5579. Paired tests do not establish a significant difference between the hybrid and IBM1 at the 0.05 level, although reranking substantially improves over its neural first stage. Under pair-specific training, LoRA projection with CSLS also performs strongly on Khmer-Vietnamese and Lao-Vietnamese, reaching 80.14% and 84.58% Recall@1. Overall, the results show complementary strengths between lexical alignment and lightweight neural adaptation in low-resource bitext retrieval. Data and code are available at https://github.com/NgThTy/translation-vn-bahna.

When an LLM judge only has to assign a three-way support label to a candidate answer given a reference, does asking it to decompose the answer into atomic claims help, and at what cost? We compare four single-call designs that share the judge model, the inputs, and the level of instruction detail: candidate-side atomic decomposition, a matched holistic rubric, reference-side decomposition that checks whether each reference claim is covered, and a bidirectional combination. Support labels are constructed from TruthfulQA, ASQA, and QAMPARI references (200 questions and 400 rows per dataset). All four designs are run with Opus-4.6, GPT-4.1, and Gemini Flash Lite; Sonnet-4.6 is added for the candidate-side and holistic designs. Candidate-side decomposition is weak where the label depends on completeness: the holistic rubric is more accurate on ASQA and QAMPARI for every judge while using fewer tokens. Reference-side decomposition is 12.5-21.3 points more accurate than holistic on ASQA for 28-29% more tokens, and stays near the holistic ceiling on QAMPARI at 63-69% more. On TruthfulQA misconceptions, candidate-side decomposition is competitive and significantly better for two judges. On a 60-row single-author subset whose labels are looser than strict reference completeness, candidate-side decomposition edges ahead of holistic for every judge, reversing the construction-label order. What a judge decomposes should follow what the label measures.

Hate speech classifiers often misclassify counter speech, including condemnation and quotation of hateful content, as hateful because they fail to distinguish propositional content from speaker stance. We identify this issue as content stance conflation and address it as a representation learning problem. We learn stance invariant representations of the same proposition by constructing contrastive pairs that share content but differ in stance. We implement this idea through factorized annotation, stance controlled augmentation, and Stance Contrastive Learning (SCL). On the Implicit Hate Corpus, stance variant augmentation reduces counter speech false positives by 60%, and SCL further improves cross domain F1 by up to 6.5 points on top of the augmented signal. On SBIC, where use mention conflation is less prominent in the training labels, SCL maintains a low counter speech false positive rate. Semantic similarity pairing on the same augmented data yields no cross domain gain over the model trained without contrastive learning (0.623 vs. 0.625), which identifies the pair construction signal rather than contrastive learning itself as the determining factor.

Speech-based Alzheimer’s disease (AD) detection increasingly relies on speech-enhanced and curated versions of the Pitt Corpus, where speech enhancement, sample selection, and demographic balancing are often treated as beneficial preprocessing steps. However, whether these transformations improve real-world AD detection or instead affect model generalization and prediction behavior remains unclear. In this work, we revisit the role of speech preprocessing and dataset curation across widely used benchmarks for speech-based AD detection. We evaluated the speech quality of different datasets, the cross-dataset generalization of multiple deep learning models under matched and mismatched enhancement settings, and the behavior of several recent large audio-language models (LALMs). Experimental results show that across multiple supervised speech models, speech-enhanced datasets often improve in-domain performance while reducing robustness in cross-domain evaluation. Matched enhancement between training and test data alleviates, but does not eliminate, this degradation. LALMs show a similar sensitivity: enhanced datasets induce stronger class imbalance and prediction shifts than unprocessed data. These results suggest that speech preprocessing and dataset curation can substantially influence downstream AD detection behavior, indicating that "cleaner" speech datasets are not necessarily more reliable for real-world AD detection.

Patient-facing medical LLMs and agents increasingly answer symptom questions before clinician contact, where the key safety question is what action the user should take next. We introduce CARE-Bench, a source-grounded benchmark that evaluates sequential patient-facing triage as a four-label per-turn current-action task. CARE-Bench contains 500 cases and 1,059 evaluated patient-disclosure prefixes reconstructed from medical dialogue, consultation, and follow-up-question sources. We evaluate 11 models on 269 held-out rounds under unprompted and minimally prompted open-ended protocols, using a fixed gpt-5.5 mapper to code each response into the four-label action space. Unprompted macro-F1 remains low, ranging from 31.2 to 50.4. Prompting improves 10 of 11 models, with prompted macro-F1 ranging from 46.9 to 63.4, but substantial threshold errors remain. Prompted models often recommend care before needed clarification is obtained; when the correct action was to ask for more information, only 33.5% of prompted outputs preserved the step. The persistence of these errors after prompting suggests that patient-facing triage is not a simple prompting problem and supports explicit evaluation of action timing before deployment.

Tang Dynasty poetry (618–907 CE) represents one of the most significant corpora in Chinese literature, yet no openly accessible, computationally-suitable resource exists with comprehensive author metadata. We present OpenQTS, a complete machine-readable edition of the Quan Tang Shi (全唐詩), containing 48,300 poems from 2,602 authors across 900 volumes. Unlike existing resources, OpenQTS provides systematic biographical metadata through algorithmic inference and authority file integration. Each poem receives a unique identifier, with proper handling of multi-part poems (15.5% of corpus) and collaborative works. All metadata includes provenance tracking, documenting whether information derives from Wikidata, biographical text inference, or historical markers.

Geo-spatial and geo-temporal reasoning require models to determine not only what information is present but also where and when it applies, which is critical for applications such as crisis response, environmental monitoring, and urban planning. This review surveys recent advances in vision–language models (VLMs) and large language models (LLMs) that address geographically and temporally grounded reasoning across tasks including image geolocation, spatiotemporal question answering, language-guided navigation, and Earth observation. We organize the literature through a structured analysis covering task families, input modalities, model architectures, and reasoning paradigms, highlighting how perception, retrieval, tool use, and intermediate reasoning steps contribute to system performance. The survey further synthesizes existing datasets and benchmarks, identifying common evaluation pitfalls such as spatial granularity mismatch, temporal drift, geographic bias, and potential data leakage. Finally, we outline key research challenges and opportunities for developing robust, verifiable, and time-aware geo-spatial AI systems while addressing broader concerns related to fairness, bias, and location privacy.

Large language models (LLMs) reason fluently but do not regulate their reasoning: they apply uniform scrutiny to every input, which leaves them vulnerable to adversarial and counterfactual prompts, while indiscriminate critique over-corrects answers that were already sound. We propose MetaCrit, a multi-agent framework grounded in Nelson and Narens’ metacognitive regulation theory that calibrates how much critique each task receives. MetaCrit separates regulation into four agents: an object-level generator, a monitoring agent that assesses response validity, a control agent that critiques logical soundness, and a meta-level synthesizer that reconciles their signals into a regulated response. Adaptivity here is input-conditioned intervention strength within a fixed pipeline: all four agents run on every input and what varies is the direction and magnitude of the correction they produce, not which stages execute. Across reasoning, safety, and bias benchmarks, MetaCrit improves truthfulness and logical soundness and reaches zero toxicity on BOLD and HONEST without a reasoning trade-off, whereas the same critique applied indiscriminately degrades performance. The cost is four calls per query, about one sixth of the cost of a dedicated reasoning model of similar accuracy. A writing study shows that MetaCrit is preferred for critical-thinking support, and its agents transfer to existing frameworks without architectural change. Code is available at https://github.com/Paparare/EduThink4AI.

Spatial reasoning, the ability to represent and reason about spatial relations, geometry, topology, and embodied environments, is increasingly vital for evaluating and deploying large language models (LLMs) and vision–language models (VLMs). While multimodal foundation models show strong performance on spatial benchmarks, navigation, and embodied planning, much of this stems from shallow pattern matching rather than grounded spatial understanding. This review offers a comprehensive, framework-driven analysis of spatial reasoning in LLMs and VLMs, covering tasks, representations, datasets, benchmarks, architectures, training, and failure modes. We also link spatial reasoning to embodied AI, emphasizing challenges in generalization, compositionality, and grounding. Our survey unifies diverse fields, NLP, vision, robotics, and cognitive science, to guide the development of truly spatially grounded foundation models.

Large language model agents have shown impressive capabilities in interactive decision-making tasks. These agents interact with environment through intermediate interfaces, such as predefined action spaces and interaction rules, which mediate the perception and action. However, mismatches often happen between the internal expectations of the agent regarding the influence of its issued actions and the actual state transitions in the environment, a phenomenon referred to as **agent-environment misalignment**. While prior work has invested substantially in improving agent strategies and environment design, the critical role of the interface still remains underexplored. In this work, we empirically demonstrate that agent-environment misalignment poses a significant bottleneck to agent performance. To mitigate this issue, we propose **ALIGN**, an Auto-Aligned Interface Generation framework that alleviates the misalignment by enriching the interface. Implemented as a lightweight wrapper, ALIGN-generated interface achieves the alignment without modifying either the agent logic or the environment code. Experiments across multiple domains including embodied tasks, web navigation and tool-use, achieve consistent performance improvements, with a 45.67% success rate improvement in ALFWorld.

Tone languages constitute over 50-70% of the world’s languages, but the vast majority are low-resource, lacking the large transcribed corpora needed for automatic tone classification. Existing datasets are typically collected at the sentence level, whereas field linguists require fine-grained syllable-level annotations. We propose ToneCL, a lightweight contrastive learning framework for few-shot syllable-level tone classification. We simulate low-resource conditions on Mandarin and Vietnamese, limiting labeled data to tens of examples per tone class. ToneCL is pretrained on unlabeled speech with augmentations that preserve tonal identity, then fine-tuned on few-shot examples. Experiments show our method consistently outperforms baselines, achieving 91.6% on six-speaker Mandarin at 10 shots. Cross-lingual transfer is also effective: pretraining on Vietnamese and fine-tuning on Mandarin reaches 91.0% accuracy at 10 shots. Ablation confirms that frequency band rejection is the most critical augmentation.

Hadamard preprocessing is widely used in large language model (LLM) quantization to reduce activation coherence and suppress peaks. Existing methods typically rely on random sign flips before the Hadamard transform, which do not use information from the target data distribution. We propose Greedy Hadamard Alignment (GHA), a simple training-free method to build a signed-permutation preconditioner from calibration data. GHA uses a pooled representative activation, a greedy search over flat targets, and an efficient alignment step that is analytically optimal once the target is fixed. We also study when one shared GHA preconditioner can transfer across LLM activations. GHA works best when activations are directionally aligned and stable across calibration sources. Overall, GHA improves post-quantization model quality over randomized Hadamard baselines, with the strongest gains in KV-cache quantization, more moderate gains for hidden states, and little compute overhead during inference.

Real-world documents are visually rich and structurally organized, with text, tables, figures, and captions interleaved across pages. Although retrieval-augmented generation (RAG) is widely used for document question answering, existing pipelines often treat such documents as fixed page- or chunk-level collections. This document-agnostic construction either dilutes evidence in dense pages or fragments semantically related layout elements. Moreover, cross-page questions often require iterative evidence exploration, but existing systems typically follow fixed retrieval paths that do not adapt to the evidence already found.

We propose DAMG-RAG, a Document-Aware Multi-Granularity RAG framework for visually rich document understanding. During ingestion, DAMG-RAG constructs layout-aware atomic retrieval units and links them to page- and section-level nodes, forming a multi-granularity document tree. During inference, a state-aware agentic workflow maintains explored regions and validated evidence, dynamically selecting exploration and navigation tools over this tree. On MMLongBench, DAMG-RAG obtains 64.7% accuracy with Claude Sonnet 4.6, yielding a 3.6-point gain over the best-performing agentic baseline in our experiments while using 18.8% of its token budget.

Recent LLM-for-Space systems address mission planning, scheduling, operations support, simulator control, and autonomy, but their evaluations use different task contracts, control settings, simulators, and success criteria. We introduce AstroAgentBench, a seven-family benchmark for executable space mission planning in the domains of scheduling, observation planning, constellation design, and relay support. For each case, an agent submits a planning artifact that is checked by an external verifier for schema, timing, geometry, resources, and mission value. Results report validity and normalized scores, with comparisons to task-specific solver references. Across five LLM agent systems and 35 held-out cases, the strongest systems approach or exceed solver-reference scores on several families, while weaker systems often fail to produce high-value valid plans and even strong systems lose quality on geometric, product-level, or design-heavy tasks. Trace analyses separate two failure points: task-contract misformulation and weak solution construction. Successful runs instead calibrate agent-written implementations against verifier feedback and adapt search to case-specific structure. Ablations show that procedure injection and memory accumulation help selectively, when they supply the missing formulation, calibration, or search support.

Large language models (LLMs) generally rely on sampling temperature to regulate the amount of randomness in their outputs. Mainstream methods for selecting temperature typically involve fixed, heuristic global settings and context or token-level calibration, rather than a geometrically interpretable representation of the communicative scenario. In this work, we introduce a simple and interpretable temperature selection framework, the **Scenario-Actor**, which adapts decoding to the inferred situational characteristics. Scenario-Actor extracts scenario-relevant structures from Transformer hidden states and determines the temperature from scenario embedding, a structured geometric representation of the communicative setting. Empirical results show that on Llama-3.1-8B-Instruct, Scenario-Actor achieves strong goal-completion performance in interactive social scenarios, significant gains on open-ended instruction following, and competitive performance on multi-turn conversation compared to multiple baselines.

Open-weight large language models face a low-cost white-box threat from representation engineering attacks. Attackers can estimate refusal directions and search for projection-matrix edits that suppress safety alignment while preserving general capabilities, within minutes on a single GPU and without gradient-based training. We propose Bait-and-Recover, a weight-level defense that places a bait adapter where attackers read activations and a paired recovery adapter at the subsequent layer. Trained via gradient routing, this decouples the observation path from the behavior path. By actively poisoning the residual signal used for measurement, Bait-and-Recover disrupts the attacker’s edit search, while the recovery layer restores clean downstream computation. Across four open-weight models, our defense raises the minimum refusal rate against white-box edit searches from 16.25% to 71.75% under a strict behavior-preservation budget (KL ≤ 0.10), with negligible impact on general benchmarks. By invalidating the core measurement assumption of these attacks, observation-path poisoning offers a practical complement to behavior-level safety training.

As text-to-motion (T2M) models move beyond short action captions toward real-world applications, they are increasingly expected to follow diverse instructions. However, existing evaluation protocols remain insufficient for systematically assessing instruction following, physical plausibility, and fine-grained controllability under such diverse language instructions. To address this gap, we introduce T2MBench, a benchmark with 1,025 prompts across four categories—Dynamics, Complexity, Interaction, and Accuracy—covering 14 representative T2M models and three evaluation dimensions: LLM-Based Evaluation, Multi-Factor Motion Evaluation, and Fine-grained Accuracy Evaluation. Our results reveal substantial gaps in current T2M models’ ability to consistently follow diverse language instructions while maintaining motion plausibility and fine-grained control.T2MBench establishes a reusable diagnostic benchmark for advancing research toward more reliable T2M generation under diverse language instructions. Data and materials are available at https://github.com/Courteous121/T2MBench-release.

Conversational multimodal emotion recognition MER requires reliable prediction when language acoustic or visual observations are missing or unreliable. Many missing-modality methods reconstruct absent inputs yet such recovery can be non-unique in dialogue context and nonverbal cues may conflict with the target utterance. To this end we propose CoRe-KD Complete-view Reference-guided Knowledge Distillation a state-anchored conflict-regularized complete-view distillation framework for robust conversational MER. A complete-view teacher provides structured references including prediction-level references fused states and modality-specific states. Complete-view State Anchoring CSA aligns incomplete-view student predictions and states with these references while Nonverbal Conflict Exposure NCE trains on target-preserving nonverbal conflict views to reduce donor-label bias. Experiments on IEMOCAP and MELD with CMU-MOSEI as a supplementary utterance-level check show consistent gains under fixed- and random-missing protocols. Comprehensive ablation studies and further analyses support the role of CSA and the complementary effect of NCE.

Sentiment analysis during political crises is difficult because language, context, and public reaction can change rapidly. This challenge is especially underexplored for low-resource languages such as Bangla. We introduce **UnrestSent200K**, a Bangla crisis sentiment dataset containing approximately 200K Facebook and YouTube comments from the July–August 2024 Bangladesh uprising. The dataset covers five event-aligned phases, ranging from early escalation and internet blackout to regime transition and a later flood crisis. Each comment is linked to its parent post, allowing evaluation both with and without discourse context. All comments are annotated through a fully human process involving 14 native Bangla-speaking annotators and senior validation. The annotation process achieves substantial agreement, with Cohen’s kappa of 0.73, Krippendorff’s alpha of 0.71, and 94.2% blind-audit agreement. We benchmark fine-tuned encoders, prompted LLMs, and LoRA-tuned LLMs. Results show that adding parent-post context consistently improves performance, while temporal shifts across crisis phases cause large performance drops. Strong LLMs perform well, but they still struggle with sarcasm, implicit political references, and meanings that depend on the event phase. UnrestSent200K provides a benchmark for studying context-aware and temporally robust sentiment analysis in low-resource crisis discourse.

Safety alignment of small language models (SLMs) typically incurs an alignment tax: benign prompts are refused along with truly harmful ones. The problem is sharper for Korean SLMs, where limited capacity meets a language whose intent signal is blurred by honorifics. We propose KoSCoPe, which combines a Hierarchical Safety Curriculum, a level-conditional adaptive margin (HARM-DPO), and Counterfactual Consistency Regularization, with a reasoning internalization schedule that recovers 1.5 to 3.2× inference speed. Across five Korean SLMs of 1.7B to 4B, KoSCoPe achieves the strongest Safety/Over-refusal trade-off among the baselines under our matched protocol and remains competitive with general-purpose models several times larger. Under paired-bootstrap tests, its Safety gains over the matched preference-optimization baselines survive multiple-comparison correction for all but the closest competitor, Cat-DPO, where the gain is smaller but directionally consistent. Preliminary cross-lingual validation on Japanese, Mandarin Chinese, and English suggests the same design can transfer beyond Korean, with larger gains where surface politeness markers are more salient.

Large language models show potential for scalable mental-health support by simulating Cognitive Behavioral Therapy (CBT) counselors. However, existing methods often rely on static cognitive profiles and omniscient single-agent simulation, failing to capture the dynamic, information-asymmetric nature of real therapy. We introduce CCD-CBT, a multi-agent framework that shifts CBT simulation along two axes: 1) from a static to a dynamically reconstructed Cognitive Conceptualization Diagram (CCD), updated by a dedicated Control Agent, and 2) from omniscient to information-asymmetric interaction, where the Therapist Agent must reason from inferred client states. We release CCDChat, a synthetic multi-turn CBT dataset generated under this framework. Evaluations with clinical scales and expert therapists show that models fine-tuned on CCDChat outperform strong baselines in both counseling fidelity and positive-affect enhancement, with ablations validating dynamic CCD guidance and diagnostic analysis supporting asymmetric interaction design. Our work offers a new paradigm for building theory-grounded, clinically-plausible conversational agents.

Legal citation in common-law systems depends not only on factual similarity, but also on the legal principle for which a precedent is invoked. However, existing benchmarks for legal citation retrieval use case facts, citation context or full judgments as inputs, where the governing legal principle is often missing or only implicitly expressed and entangled with broader context. As a result, models may retrieve precedents that are factually similar yet doctrinally irrelevant. This limitation is particularly consequential in Singapore, where the legal system has evolved independently: only domestic precedents are binding, while foreign authorities serve merely as persuasive references. Thus, we propose a new retrieval paradigm that ranks cited cases based on queries integrating case facts and explicit legal principle, inspired by the real-world legal reasoning workflows. To support this paradigm, we further introduce SG-LegalCite, a dataset of 100,890 case–principle pairs extracted from 8,523 Singapore Supreme Court judgments spanning 2000–2025. Experiments across 11 baselines demonstrate the effectiveness of our principle-augmented retrieval paradigm, showing that explicit legal principles provide strong discriminative signals for legal citation retrieval. Our dataset and code is at https://github.com/ShannonLeeYuehErn/SG-LegalCite

Fine-tuning has emerged as a widely adopted approach for adapting Large Language Models (LLMs) to a variety of downstream tasks. However, how it reshapes their internal mechanisms remains poorly understood. To address this, we investigate how fine-tuning alters internal representations in LLMs, including attention patterns and layer-wise activations, and examine whether these changes are linked to task-relevant components identified by Edge Attribution Patching (EAP) (e.g., attention heads and logit-level activations) that drive task performance. We find that EAP-identified components are concentrated within specific layers, indicating a degree of functional localisation in how models internalise task-specific behaviour. Notably, the distribution of these components across layers is largely uncorrelated with the layers undergoing the most substantial representational changes during fine-tuning. Furthermore, we observe that overlap in EAP-identified components across tasks does not translate into cross-task performance transfer if the tasks are different in nature (e.g., classification vs. generative tasks). More specifically, fine-tuning on one task can lead to a degradation of performance on another when the two tasks exhibit a high degree of overlap in their EAP-identified components.

Robust detection of AI-generated text under deployment conditions is challenging: distribution shifts across domains and generators, adversarial perturbations of the input surface, and the absence of target-domain labels for threshold calibration all degrade detectors that perform well in-domain. We present DeBERTa-ConPara, a deployment-oriented detector combining attack-aware Unicode preprocessing with a contextual transformer encoder trained over HC3 Plus, M4, MAGE and RAID. Our central finding is that preprocessing acts in opposite directions depending on where it is applied: normalising the training corpus deduplicates it, collapsing 35.4% of RAID rows into copies of their clean siblings and deleting the adversarial supervision, whereas normalising at inference is an effective defence. A factorial varying the two placements independently identifies raw training with normalised inference as the best configuration, reaching 99.61% AUROC, 99.01% TPR@5% FPR and 96.57% TPR@1% FPR on the official RAID hidden test, alongside 93.14% average balanced accuracy across HC3 Plus and MAGE under a fixed threshold. The gain is confined to two of twelve attack classes: homoglyph and zero-width-space insertion rise from 11.05% and 1.12% to 96.98%. The same signature reproduces in a zero-shot detector of different architecture, showing the effect belongs to the attacks rather than to our model. We additionally report two negative results: semantic-invariance augmentation through paraphrasing and supervised contrastive learning (ConPara) does not improve the best configuration, and the handcrafted feature-fusion branch is inert in distribution and harmful outside it.

Sight-singing is a fundamental yet challenging task in music education, where timely and fine-grained feedback is crucial for helping learners identify mistakes and improve their performance efficiently. However, existing automatic methods mainly focus on isolated subtasks such as alignment or scoring, and are unable to provide end-to-end diagnostic feedback with explicit identification of pitch, rhythm, and musical fluency errors. In this paper, we propose Cantus, a novel multimodal large language model framework for sight-singing assessment and exercise recommendation generation. Cantus jointly models vocal audio and symbolic musical representations, and leverages a pedagogically grounded error representation module together with a large language model to diagnose performance errors, generate structured assessment comments, and provide personalized exercise recommendations. To support this task, we construct a new dataset MIVO (MIDI-Voice) containing 1,003 real-world sight-singing samples with aligned MIDI notes and vocal recordings collected from authentic teaching scenarios. Experimental results demonstrate the effectiveness of Cantus in sight-singing assessment, achieving strong performance across multiple metrics. Human evaluation of feedback quality shows the potential of Cantus in supporting student-centered sight-singing learning.

General Retrieval-Augmented Generation (RAG) systems would suffer from performance degradation in multilingual scenarios due to the complexity of integrating different retrieval results from multilingual collections. In this paper, we propose CoSelector, a multi-agent collaborative framework for multilingual retrieval selection, which is optimized by the centralized training with decentralized execution. Under this paradigm, agents share the same goal but execute decisions using only local observations. CoSelector consists of a Selector agent and a Generator agent. The Selector dynamically filters and composes the most pertinent multilingual documents into a coherent context, while the Generator processes this context to understand and integrate information for answer generation. Across the 15 languages evaluated on MKQA and XOR-TyDi QA, CoSelector achieves competitive improvement in response language consistency, demonstrating its effectiveness in both linguistic alignment and the selection of retrieved results.

Long-horizon multimodal agents depend on external memory. However, similarity-based retrieval often surfaces stale, low-credibility, or conflicting items, which can trigger overconfident errors. We propose Multimodal Memory Agent (MMA), which assigns each retrieved memory item a dynamic reliability score by combining source credibility, temporal decay, and conflict-aware network consensus, and uses this signal to reweight evidence and abstain when support is insufficient. We also introduce MMA-Bench, a programmatically generated benchmark for belief dynamics with controlled speaker reliability and structured text–vision contradictions. Using this framework, we uncover the "Visual Placebo Effect," revealing how RAG-based agents inherit latent visual biases from foundation models. On FEVER, MMA matches baseline accuracy while reducing accuracy standard deviation by 35.2% and improving selective prediction utility; on LoCoMo, a safety-oriented configuration improves actionable accuracy and reduces wrong answers; on MMA-Bench, MMA reaches 41.18% Type-B accuracy in Vision mode, while the baseline collapses to 0.0% under the same protocol. Code is available at https://github.com/AIGeeksGroup/MMA.

Language models are increasingly expected to support users in stressful or uncertain situations, where warmth and empathy are desirable. While recent work links warmer models to lower factuality and reliability, we show that this conclusion is incomplete: prior studies often use warmth as a global persona and conflate tone effects with SFT data distribution shift. We revisit this issue through prompting, fine-tuning, factual robustness, and empathy evaluation, using an 11-dataset benchmark suite and EmoFact, a benchmark for factual accuracy under emotional context. Across three large language models, we find that warmth effects are task- and model-dependent, and that task-aligned adaptive tone can improve empathy while preserving reliability. We further train human-validated empathy evaluators to evaluate the output of our finetuned model. Our results suggest that models can be made more supportive in human relationships and social interactions without compromising overall reliability.

Knowledge graph question answering (KGQA) is a promising approach for mitigating LLM hallucination by grounding reasoning in structured and verifiable knowledge graphs. Existing approaches fall into two paradigms: retrieval-based methods utilize small specialized models, which are efficient but often produce unreachable paths and miss implicit constraints, while agent-based methods utilize large general models, which achieve stronger structural grounding at substantially higher cost. We propose RouterKGQA, a framework for specialized–general model collaboration, in which a specialized model generates reasoning paths and a general model performs KG-guided repair only when needed, improving performance at minimal cost. We further equip the specialized model with constraint-aware answer filtering, which reduces redundant answers. In addition, we design a more efficient general agent workflow, further lowering inference cost. Experimental results show that RouterKGQA outperforms the previous best by 3.57 points in F1 and 0.49 points in Hits@1 on average across benchmarks, while requiring only 1.15 LLM calls per question on WebQSP (1.23 on CWQ). Code and models are available at https://github.com/Oldcircle/RouterKGQA.

Memory-augmented assistants must answer questions from long-horizon conversations in which facts are not only missing, but also revised, corrected, or mutually inconsistent. Existing methods largely optimize retrieval, storage, or updating, and many controllers still treat failure primarily as insufficient evidence. We argue that conversational memory QA also requires an explicit model of inconsistency: when remembered claims disagree, a system should not simply retrieve more or guess, but may need to resolve versions, return a qualified answer, ask for clarification, or abstain. We introduce CoMMA, a transparent read-time controller with explicit conflict state for conversational memory QA. CoMMA extracts provenance-aware claims from retrieved memories, groups potentially incompatible claims into conflict clusters, infers lightweight question intent, and chooses among direct answer, qualified answer, clarification, and abstention. In an evaluation centered on the full LongMemEval-Oracle benchmark, a LongMemEval conflict-focused diagnostic slice, and FactConsolidation-20, CoMMA performs best on the main LongMemEval answer-quality comparison, is the only method with non-zero conflict F1 on the conflict-focused diagnostic slice, and ties the strongest write-time overwrite baseline on FactConsolidation-20. These results suggest that making conflict explicit at read time improves both decision quality and the auditability of memory-aware behavior.

Safety evaluations of large language models (LLMs) have largely focused on whether models refuse explicitly harmful requests. However, in high-stakes decision-support settings, seemingly legitimate requests may conceal missing information or hidden risk signals. We propose LogicLure, a benchmark for evaluating whether LLMs avoid premature decisions and actively seek evidence under such plausible scenarios with hidden risks. The benchmark spans eight domains and pairs persuasive visible narratives with hidden evidence maps accessible only through verification. Our evaluation measures not only the decision of models, but also whether they ask verification questions and whether their final decision is grounded in the revealed evidence. In our evaluation, the strongest model reaches only 26.4% evidence-grounded refusal despite 94.7% overall refusal. Furthermore, we analyze how scenario surface cues, explicit scrutiny setting, and prompt-induced stance shape these decisions. Together, these analyses characterize the key behavioral factors underlying LLM performance on LogicLure. The benchmark and code will be made publicly available.

Additive quantization enables extreme LLM compression while preserving lookup-table inference with free-form codebooks. At 2 bits per weight, however, greedy sequential initialisation can cause large quality losses that wider assignment search and PV-tuning do not fully erase. We show that this failure reflects an interaction between representational pressure and sequential codebook coupling. We use 𝜌=N/K^M as a structural pressure measure and propose OA-EM, a hard generalised-EM initialisation procedure using Hessian-weighted Mahalanobis distance to minimise output-aware reconstruction error. Across four models from 3B to 14B parameters, OA-EM consistently lowers matched post-PV perplexity. Controlled experiments at a fixed bitrate show that 𝜌 alone does not explain sensitivity to initialisation; the number and size of the additive codebooks also matter. A Euclidean-EM ablation and curvature diagnostics show that the gain is not explained by EM refinement alone: OA-EM reduces error in high-curvature directions. On Llama 3.2 3B, OA-EM also improves the measured quality-compute trade-off. Overall, the final quantized solution remains persistently dependent on initial codebook geometry. Code: https://github.com/kenno94-IK/aqlm-oaem.

We present FrenchNews-7, a cross-publisher France-based French-language news editorial desk classification benchmark combining a large multi-outlet corpus, a URL-derived seven-class taxonomy, and a fine-tuned CamemBERT classifier. Labels are assigned via a hybrid pipeline combining publisher URL slugs with LLM annotation for structurally ambiguous cases, audited through an inter-rater study (2 humans + 2 LLMs; pairwise κ ≥ 0.766, human–human κ = 0.806). We evaluate lexical, multilingual, and French-specific trained classifiers under both in-distribution and held-out-publisher settings, with additional comparison against zero-shot LLM baselines (GPT-OSS-120B, Mistral Small 3.2, Llama-3.3-70B) on the held-out pool. The strongest model, CamemBERT-base on full article text, outperforms headline-only input, generalizes to unseen outlets, and exceeds all three zero-shot LLM baselines on overall recall (0.799), with the gap concentrated in the ambiguous editorial-boundary categories Économie and Société. Cross-publisher evaluation reveals uneven boundary stability: Sport, Culture Loisirs, and International transfer cleanly, while Économie (recall = 0.517) is statistically indistinguishable from blinded human agreement on the same articles (0.55, n = 40), and Société (precision = 0.577) absorbs boundary ambiguity, both suggesting editorial conventions rather than recoverable classifier headroom. The fine-tuned CamemBERT-base model is released on Hugging Face (https://huggingface.co/LeFrenchNewsLab/camembert-base-frenchnews7) and the labeled manifest on (https://huggingface.co/datasets/LeFrenchNewsLab/frenchnews-7), alongside reference collection scripts to assist re-fetching and a reliability-tier guidance table.

We introduce SweSpecQA, to our knowledge, the first Swedish benchmark for specialized institutional and professional QA with paired text and full-page PDF evaluation. It contains 1,500 questions from public Swedish examination and certification materials across six domains: EU aptitude, boating, antibiotic stewardship, interpreter licensing, physician licensing, and pharmacist licensing. Of these, 234 questions require an original visual or layout-dependent source element; the remaining 1,266 are text-eligible. All benchmark questions are drawn directly from Swedish-language source materials, avoiding translation artifacts in multilingual adaptations of English benchmarks. We evaluate 17 models under a closed-book protocol, reporting accuracy by domain, exam subtype, visual/layout dependence, modality, and prompting strategy, as well as whether models follow the requested answer format. On the text-eligible subset, every multimodal model performs worse with full source-page input than extracted text; PDF penalties range from 4.0 to 38.2 points under direct prompting and from 1.7 to 36.4 points under chain-of-thought prompting. Reasoning prompts often improve PDF performance for several open-weight VLMs, while sometimes reducing clean-text accuracy. These results show that document-native evaluation exposes failures hidden by text-only evaluation and that preprocessing remains important for reliable Swedish exam-style QA.

Small language models (SLMs) are efficient and scalable, but their multilingual capabilities degrade severely at sub-billion scales, especially for Southeast Asian (SEA) languages. We introduce DuDi, a dual-signal multilingual distillation framework that combines an online sequence-level signal with off-policy and on-policy token-level signals. DuDi further uses a cross-lingual verbalizer to refine teacher feedback and improve teacher-student transferability in multilingual settings. Experiments on SEA-HELM across multiple model families, scales, and teacher-student settings show that DuDi consistently outperforms competitive distillation baselines. Ablations and analyses confirm that sequence-level optimization, token-level supervision, and cross-lingual verbalization provide complementary and transferable learning signals for multilingual SLMs.

Frontier LLMs perform well in Western contexts, but remain poorly tested on underrepresented cultures such as those in Southeast Asia (SEA). Existing NLI benchmarks are largely Western-centric, translation-derived, or monolingual, limiting their ability to measure culturally grounded reasoning. We introduce SEA-NLI, a native, culturally grounded NLI benchmark covering eight SEA countries in English and native regional languages, verified by native speakers. Across 17 encoder and decoder models, we observe a low performance from all models, especially for knowledge-intensive categories such as Languages and Science and Technology. Our analysis shows that failure cases mainly stem from missing SEA cultural knowledge: SEA-adapted models and culture-aware prompting improve performance, while CoT prompting offers limited gains.

Large Language Model (LLM) personalization is challenging because user preferences are rarely stated explicitly; instead, they emerge as implicit, multifaceted signals in users’ past interaction histories. Existing methods often compress user histories into a single representation or retrieve and inject past records without explicitly linking them to supporting evidence, which can lead to hallucinated or overly generic outputs. In this work, we propose Multi-Aspect PersonaLized Modeling with Evidence (MAPLE). MAPLE structures implicit user preferences through a MULTI-ASPECT PREFERENCE PROFILE and connects them to target-specific grounding evidence through a PERSONALIZED EVIDENCE PLAN. By selecting only generation points that are both relevant to the user and supported by available evidence, MAPLE improves personalization and grounding faithfulness simultaneously. Experiments on the Games and Books domains with two backbone LLMs show that MAPLE consistently outperforms strong baselines and generalizes robustly under limited-history and unseen-user settings.

While in-context machine translation approaches lead to notable progress for low-resource languages, it is not yet clear what kinds of linguistic information in prompts can consistently and reliably lead to performance gains. We evaluate whether English semantic representations, specifically Abstract Meaning Representation (AMR) and its extension Uniform Meaning Representation (UMR), can improve LLM-based MT via prompting. Across 12 languages, 7 models from 5 families, and multiple prompting strategies, we find that conclusions about their effectiveness are highly sensitive to the setting. We further observe that prompts using UMR tend to outperform those using AMR, particularly in few-shot scenarios and for certain models. These results highlight the need for comprehensive evaluation for in-context learning work.

Sentence encoders are foundational to a wide range of NLP applications, yet they are predominantly evaluated through performance on downstream tasks. Such task-specific evaluations provide limited insight into whether the resulting sentence representations capture fundamental compositional relationships in a task-independent manner. To address this gap, we introduce a set-theoretic framework for evaluating the compositional properties of sentence embeddings. Drawing on classical set theory, we define six evaluation criteria based on three core set-like text operations: TextOverlap, TextDifference, and TextUnion. We use these criteria to systematically evaluate 19 sentence encoders, including 7 classical models and 12 Large Language Model (LLM)-based encoders. Our results reveal substantial differences in their compositional behavior: notably, SBERT exhibits the strongest overall alignment with the proposed set-theoretic criteria, outperforming even recent LLM-based encoders. We further examine whether these intrinsic compositional properties are associated with downstream performance across five NLP tasks. Among them, summarization exhibits the strongest correlation with the TextOverlap and TextDifference criteria, suggesting that our framework may also provide useful signals about downstream capability without relying exclusively on task-specific evaluation. Finally, we release a benchmark comprising ~192K samples to support systematic evaluation and future research on the compositional properties of sentence embeddings.

Reasoning Vision-Language Models (VLMs) achieve strong performance on complex multimodal tasks, but reliable real-world application requires handling visual inputs that are messier than clean, curated benchmarks. Existing works mainly evaluate such reliability of VLMs through input corruptions, such as noise, blur and weather effects, which make visual evidence harder to perceive. This leaves a critical reliability failure mode underexplored: a model may perceive the evidence correctly, yet reason from plausible but irrelevant and distracting evidence and propagate this mistake to its final answer. To address this gap, we introduce **Distract-Bench**, a benchmark for evaluating VLM robustness to **semantic visual distractions**, defined as meaningful but task-irrelevant visual cues added to inputs while preserving the ground-truth answer. We comprehensively evaluate eight leading open-source and two closed-source VLMs across conventional vision corruptions and Distract-Bench. Our results show that Distract-Bench exposes a robustness failure distinct from vision corruptions: reasoning VLMs largely track their non-reasoning base models under perceptual degradation, but show consistently lower robustness to semantic distractions. Further analysis shows that these distractions often enter the reasoning process of VLMs, are treated as evidence, and lead to incorrect answers. Together, these findings reframe robustness evaluation for reasoning VLMs, shifting the focus from degraded perception to distractions for reliable real-world visual reasoning. Our data and code are available at _https://github.com/Yizheng-Sun/Distract-Bench_.

Multimodal Large Language Models (MLLMs) struggle with continual learning, often suffering from catastrophic forgetting when adapting to sequential tasks. We introduce a routing-based architecture that integrates new capabilities while robustly preserving foundational knowledge. While Multi-Task Learning (MTL) is a strong performance reference, its training cost grows linearly with the number of tasks. In contrast, our method maintains fixed per-task training data and compute requirements regardless of the task sequence length. Across models ranging from 2B to 8B parameters, we demonstrate that our routing approach performs on par with MTL while retaining the training efficiency of sequential fine-tuning. Beyond merely mitigating forgetting, we observe that token-level routing facilitates cross-modal transfer, leveraging knowledge from one modality to bolster performance in another. We also show that routing remains robust even with large expert pools and effectively capitalizes on task relatedness. Finally, we show that our method scales favorably, with larger models exhibiting minimal degradation compared to fully specialized fine-tuning.

When a language model ignores retrieved evidence and answers from memory instead, this is not a knowledge failure, the correct information is present. It is a routing failure; the model commits to its parametric answer despite the evidence signal. We show this distinction is mechanistically meaningful, localizable, and correctable at inference time without training data or weight modification.

A single prefill-margin computation detects routing failures (AUC ≥ 0.93 across five model dataset pairs) and gates a format-appropriate logit-bias correction. On MedQA multi-option conflicts, this achieves 77.5% restoration at zero harm, exceeding the strongest prompting baseline by 67.5 pp and matching domain-matched ITI without training data. On open-ended PopQA (3,141 items, three models), balanced alias promotion achieves 80-88% restoration at near-zero harm, Pareto-dominating prompt-only baselines in five of six cross-model conditions. The central mechanistic finding is that effective late-layer intervention sites are structurally distinct from the causal commitment locus (a five-head cluster, joint flip rate 88.6%, p < 10^-9), suggesting that steering can succeed through residual-stream write access without exactly identifying the full decision circuit.

Prompt-based prediction pipelines suggest that large language models (LLMs) perform poorly on time-series classification, raising doubts about whether they encode meaningful temporal structure. We show that this conclusion reflects limitations of prompt-based generation rather than the model’s representational capacity by directly comparing prompt outputs with linear probes over the same internal representations while controlling for important confounding factors. While prompting performs near chance, linear probes improve average F1 from 0.15–0.26 to 0.61–0.67, often matching or exceeding specialized state-of-the-art (SoTA) time-series models. Layer-wise analyses further show that class-discriminative time-series information emerges in early transformer layers and is amplified with multimodal inputs and the presence of pretrained weights. Together, these results demonstrate a systematic mismatch between what LLMs internally represent and what prompt-based prediction pipelines reveal, leading current prompt-based prediction pipelines to underestimate the degree to which these models function as useful time-series feature extractors.

Medical vision-language models (VLMs) commit clinical errors when generating chest X-ray reports: they fabricate findings that are not present in the image, miss important ones, or locate them incorrectly. We steer the error composition of generated reports without weight updates by decoding-time residual steering on a per-token sparse autoencoder (SAE) basis: Top-K SAEs on late layers, per-error-type causal feature screening against clinical errors, then combined suppress/boost intervention at inference time. On the MIMIC-CXR test split, our inference-only method improves the quality of generated reports for three radiology VLMs (RadVLM, LLaVA-Rad, and CheXOne), with relative improvements of +5.4%, +7.2%, and +17.0% in the clinical composite metric, and statistically significant GREEN gains on all backbones. The gain is an explicit, controllable trade-off rather than uniform error reduction: steering shifts models toward more complete reporting, substantially fewer missing findings, at the cost of some additional false findings, and we analyse and quantify this redistribution per error type. A cross-model feature alignment shows that the quality-promoting (boost) directions have functionally similar error-type profiles across architectures, whereas hallucination-linkedm (suppress) directions are model-specific. Census-motivated steering should therefore treat suppression per-backbone, rather than sharing a universal suppress list. The same recipe applied zero-shot to IU-Xray (GREEN +7.7% rel., on RadVLM) suggests that the identified features are properties of the model rather than of the training corpus. The causal feature sets and an interactive feature dashboard are shared.

This position paper argues that researchers should not only study how human biases enter large language models (LLMs), but actively leverage those biases as diagnostic tools to surface society’s "unwritten code"—the tacit norms and hidden heuristics that govern high-stakes decisions yet resist explicit articulation. We present a case study in peer review: a framework that iteratively pushes LLMs to generate self-consistent hypotheses explaining score differences across 26,731 submissions to 46 conferences. LLMs’ normative priors (cues extracted by self-talk) emphasizing theoretical rigor are systematically updated toward posteriors (cues actually used in the judgment) emphasizing storytelling and contextualization—a finding cross-validated by independent human survey evidence. Human reviewers explicitly reward prior-aligned aspects (r=0.49) but rarely articulate the storytelling posteriors (r=-0.14), despite implicitly rewarding them. These findings generalize to out-of-sample judgment and different models. We argue this framework applies to hiring, admissions, and criminal justice, offering a scalable path to making hidden evaluative codes visible for public scrutiny and equitable reform.

Estimating model performance without labeled data is important for deploying NLP systems across diverse domains. Prior work has proposed proxy measures based on distribution shift or predicted correctness, but it remains unclear whether these methods reliably preserve the ranking of domains, which is critical for model selection and risk triage. We study this problem from an ordinal perspective using a two-step framework that pairs a base classifier with an error predictor to estimate instance-level failures and disparities and aggregate them into label-free performance estimates. Across multiple sample sizes from the GeoOLID and Amazon Reviews datasets, large language model–based error predictors substantially outperform drift-based baselines in 𝜌 and 𝜏 rank correlations. We further show that lower absolute estimation error (RMSE) does not necessarily imply better ranking, highlighting the importance of evaluating ordinal consistency directly.

WARNING: This paper contains offensive and hateful speech in the appendix.

Persona-driven generations (PDGs) have seen prolific use in research and industry applications, where a large language model (LLM) takes on a “persona” while completing some task. While persona expressed through free-form text (like dialogue) has substantial work investigating stability or consistency, relatively, persona expressed in non-text-heavy outputs (like in multiple-choice question answering, or MCQA) is often overlooked. We work to address this gap, seeking to understand the instability of LLM PDGs in MCQA tasks. We develop three metrics investigating the performance, outcome, and question correctness stability, evaluating three distinct dimensions. Using these metrics, we find that instability varies consistently between model families and model size, and across question domains, with math/commonsense questions leading to greater instability. We also find task prompt format introduces more prediction instability than other hyperparameters, like temperature. Finally, we find that instability is related to task accuracy, and using our instability metrics, find different experimental settings that result in different best and worst personas for tasks, despite their similarity. This reveals the importance of checking hyperparameter instability in PDGs.

Recent work has explored the use of large language models (LLMs) to generate tutoring responses in mathematics, yet it remains unclear how closely their instructional behavior aligns with expert human practice. We analyze a dataset of math remediation dialogues in which expert tutors, novice tutors, and seven LLMs of varying sizes, comprising both open-weight and commercial models, respond to the same student errors. We examine instructional strategies and linguistic characteristics of tutoring responses, including uptake (restating and revoicing), pressing for accuracy and reasoning, lexical diversity, readability, politeness, and agency. We find that expert tutors produce higher-quality responses than novices, and that larger LLMs generally receive higher pedagogical quality ratings than smaller models, approaching expert performance on average. However, LLMs exhibit systematic differences in their instructional profiles: they underuse discursive strategies characteristic of expert tutors while generating longer, more lexically diverse, and more polite responses. Regression analyses show that pressing for accuracy and reasoning, restating and revoicing, and lexical diversity, are positively associated with perceived pedagogical quality, whereas higher levels of agentic and polite language are negatively associated. These findings highlight the importance of analyzing instructional strategies and linguistic characteristics when evaluating tutoring responses across human tutors and intelligent tutoring systems.

Traditional lossless text compression preserves every byte, but its gains on natural language are often modest. We study lossy semantic text compression, where the encoder strategically deletes parts of the text and a large language model (LLM) reconstructs the original from the retained skeleton. We benchmark a progression of deletion strategies, from character-level and word-level rules driven by word length or corpus frequency to surprisal-based and hybrid rules that require a neural encoder, and reach three findings. First, deletion by corpus word frequency is a strong low-cost baseline: a static frequency lookup remains competitive with far more expensive semantic methods. Second, semantic and hybrid methods gain most at mild-to-moderate compression, whereas at the lowest retention rate frequency-based deletion matches them and keeps more named entities. Third, QLoRA fine-tuning yields a compact local decoder that, once adapted, is competitive with zero-shot Gemini 2.0 Flash. Additional English and Chinese experiments show that the framework transfers across domains, while the best deletion rule remains dataset-dependent.

While Large Language Model (LLM) based agents excel at complex tasks, their performance in open-ended scenarios is often constrained by isolated operation and reliance on static databases, missing the dynamic knowledge exchange of human dialogue. To bridge this gap, we propose the INteractive Memory Sharing (INMS) framework, an asynchronous interaction paradigm for multi-agent systems. By integrating real-time memory filtering, storage, and retrieval, INMS establishes a shared conversational memory pool. This enables continuous, dialogue-like memory sharing among agents, promoting collective self-enhancement and dynamically refining the retrieval mediator based on interaction history. Extensive experiments across three datasets demonstrate that INMS improves agent performance by effectively modeling multi-agent interaction and collective knowledge sharing.

Agentic task-oriented dialogue (TOD) requires systems to track concurrent goals, dependencies, and long-horizon state. We examine goal-lifecycle recovery from fixed dialogue trajectories. ATOD contains 1,000 synthetic dialogues annotated for six advanced-TOD properties, and ATOD-Eval defines metrics for dependency-sensitive completion, memory recall, and proactivity. We implement a symbolic–vector memory evaluator for ATOD-Eval. Among five prompt-only predictors and six matched-backbone memory baselines, our evaluator is the only configuration above 90% in both goal detection F1 and conditional status accuracy on medium dialogues. On complex dialogues, no baseline exceeds it on both metrics; it has the highest conditional status accuracy within the memory-based block and the lowest measured per-turn latency. These experiments assess lifecycle tracking rather than interactive agent task success. A cross-family judge swap and a manual audit with 94.0% agreement provide initial checks on measurement reliability. Code and data will be released at https://github.com/amazon-science/ATOD.

Role-Based Access Control (RBAC) is fundamental in secure systems handling healthcare, financial, and government data. Prior work on LLM safety under role constraints has largely relied on fine-tuning or prompting-based enforcement. In contrast, we take a mechanistic view and probe internal representations to identify signals associated with roles, permissions, and refusal behavior. We evaluate this approach across three datasets, database- and department-separated splits, varying policy complexity levels, and adversarial prompts. On the structured Text-to-SQL datasets, simple probes achieve strong performance, reaching average macro-F1 scores of .913 on BIRD and .926 on Spider, while remaining more stable than prompting as policies grow longer. However, probing does not perform equally well in all settings: it underperforms prompting in most OrgAccess comparisons and can degrade substantially under adversarial instructions. Overall, our results show that access-control information can be strongly represented in LLM hidden states, but its usefulness depends on the model, domain, and prompt.

Multimodal emotion recognition (MER) in conversation is inherently ambiguous, as linguistic, acoustic, and visual cues may support multiple plausible interpretations. Existing approaches often treat ambiguity as prediction uncertainty rather than modeling its structured sources. We propose UAF, an ambiguity-aware framework for MER built around a single unified ambiguity signal. UAF estimates a normalized per-sample Model-based Ambiguity Degree (MAD) from the predicted emotion distribution by measuring its proximity to the maximally ambiguous uniform distribution, and uses it as a shared control signal rather than a post-hoc uncertainty score. This single scalar coordinates three parts of learning: it aligns representation radius with ambiguity, emphasizes ambiguous samples during optimization, and scales OT-based perturbation consistency. The fused evidence is organized into context, speaker, temporal, cross-modal, and residual subspaces that serve as the substrate on which this signal operates. Experiments on IEMOCAP, CMU-MOSEI, and MELD show that UAF yields consistently better calibration and high-ambiguity reliability alongside competitive accuracy and weighted-F1, showing that a single ambiguity signal can be an effective shared training signal for reliable MER

Large language models are increasingly used as math tutors, raising two questions: can a single LLM detect a student’s error, diagnose the underlying misconception, and remediate it through multi-turn dialogue—and can we trust the numbers that report how well it does? We address both with MisconceptDialogue, a fully automated framework that runs the full pipeline on five open-weight LLM tutors (7B–14B) interacting with LLM-simulated students over 5,000+ dialogues—a deliberately simulated protocol (paired conditions, a wrong-diagnosis control unsafe with real learners, scale) whose results do not establish real-student learning. On measurement, our headline finding is uncomfortable for the field: benchmark-style exact-match scoring common in standard math NLP—a lower-bound stress test—reports 42.1% measured Remediation Success Rate on identical dialogues where a human-calibrated LLM-as-judge reports 81.3%, a 39.2 percentage-point spread larger than the combined effects of tutor model, teaching strategy, and supplied diagnostic information. This is not a single-judge artifact: four independent LLM judges and 200 human annotations converge to within a few points (κ up to 0.91). On capability, the verdict is equally striking: the score barely changes whether we tell the tutor the student’s actual misconception or a deliberately swapped one—under the tested diagnosis-presentation format, these tutors show little evidence of reliably conditioning on the diagnostic information they are handed. For LLM tutoring evaluation to mature, multi-criterion scoring with multi-judge convergence and human anchoring should become standard practice—and the measurement question deserves as much scrutiny as the capability question it claims to answer.

How do interpretable internal features that implement a learned capability emerge during training? We present a mechanistic interpretability study of 3-digit addition in a 19M-parameter transformer trained from scratch, using the grokking transition as a natural microscope for feature genesis. Our framework, Causal Genesis Attribution (CGA), combines checkpoint-resolved training dynamics, Sparse Autoencoder (SAE) feature analysis, and counterfactual training interventions to trace arithmetic and carry features from emergence to causal function. We find layer-4 SAE features that are silent on non-arithmetic text, selective for the arithmetic probes, and co-emerge with validation accuracy at step 500 (near-zero to 85%). Zero-ablating the top arithmetic features drops exact-match accuracy from 100% to 0%, showing that these directions are necessary for the learned addition algorithm. A four-arm intervention study reveals what sustains these features: removing arithmetic data collapses feature activation by 81%; matched numeric-noise replacement also fails, ruling out surface token statistics; and poisoning answers by +137 hyperactivates features by 27% while making the model produce systematically shifted outputs. Across four seeds, the genesis detector fires consistently, with detection accuracy stable at 85.6% ± 1.0%. Together, these results provide a mechanistic data–feature–capability chain for an emergent arithmetic reasoning ability.

LLM benchmarks often pool rows from several sources (judges, retrieval configurations, training frameworks) and report one headline score. When label rates differ by source, a detector can earn that score by recognizing a row’s source, not its quality, and such a score is no promise about a new source. Our audit reports three AUCs of one detector: given features plus source (AUCfull, the headline), features alone, and source alone. Their ratio L=AUCsource/AUCfull shows how close the source alone comes to the headline. The precondition: L is read only above a default floor, AUCfull≥ 0.65, because two near-chance AUCs give a ratio near 1 although nothing leaks. On four RAGBench configurations (adherence rates 40.2% to 94.2%) the headline is 0.843 and the configuration label alone reaches 0.811 (L=0.96), above every per-query feature. When five LLM judges grade the same hard-math answers, L=0.84 looks similar, but the headline (0.594) fails the floor: judge identity predicts nothing. The floor, not L, separates the two panels. Six further audits with the floor and a 0.90 band on L frozen add the other outcomes, including a floor-passing panel whose source carries nothing.

Patch retrieval, the task of finding the commit that fixes a known vulnerability, is the foundation of vulnerability management workflows, yet 60% to 63% of CVEs in the major advisory databases lack a patch link. We present PatchHolmes, a two-phase patch retrieval system that pairs a hybrid first-stage retriever with an agentic second-stage inspection loop. Unlike pointwise prior work that scores each candidate independently, the Phase 2 agent reads the top-100 listwise: it sees the full candidate list at once and selectively reads 3 to 10 commits through four budgeted tools before submitting a single best commit. On GitHubAD, PatchHolmes beats the pointwise binary classifier Favia by 25.34% Recall@1 and the retrieve-and-CoT baseline IRCoT by 31.40%, at one agent conversation per CVE versus Favia’s ten; with the candidate set held identical, the agent adds 27.32% Recall@1 over taking the retriever’s top candidate, and the same agent, transferred unchanged to PatchFinder_top10, lifts Recall@1 from PatchFinder’s own top-1 pick (24.28%) to 39.86%. Swapping the LLM backbone within the Qwen family changes Recall@1 by under 1%, and a second model family (gpt-oss) stays far above the no-agent floor, so the gain comes from the listwise agent loop; the entire system runs on a frozen open-weight model over a local Git repository, without fine-tuning or external search APIs.

Constrained decoding is widely used to make large language models produce structured outputs that satisfy schemas such as JSON. Existing work mainly treats schemas as structural constraints, overlooking that schema-key tokens also enter the autoregressive context and may guide generation. To the best of our knowledge, we present the first systematic study of schema keys as an implicit instruction channel under constrained decoding. We formulate structured generation as a multi-channel instruction problem, where task signals can be placed in prompts, schema keys, or both. We further provide a projection-aware analysis that gives a sufficient condition under which an unconstrained expected-score advantage of an instructional key is preserved after grammar projection. Experiments on GSM8K and Math500 across seven language models show that changing only schema-key wording can substantially affect accuracy, with both positive and negative effects across models. Prompt-level and schema-level instructions also interact non-additively. The evidence is substantially stronger on GSM8K than on Math500. Our findings show that schema design is not merely output formatting, but part of instruction specification in structured generation.

Chain-of-thought (CoT) faithfulness is measured almost exclusively in single-turn settings, leaving open how the reasoning-trace-to-answer relationship evolves as conversations lengthen. We introduce turn-level CoT counterfactual deletion: at every assistant turn we regenerate the answer at five truncation levels of the current-turn thinking block under greedy decoding, and label a turn anchored—a descriptive label for causal redundancy of the trace, not a mechanistic claim—when all five regenerations agree on the same answer. Across 67 multi-turn sharded-GSM8K conversations from DeepSeek-R1-Distill-Qwen-7B, the anchored fraction grows 3.9× from short (< 10 turns, 7.2%) to long (> 20 turns, 28.4%). Strict early < mid < late monotone ordering holds in five of seven reasoning models from three base families and four post-training pipelines (binomial sign test, p=0.002). A supplementary pooled GEE across all seven models converges on this direction (β=1.12, p=5.1×10⁻⁸, n=2,838 turn-level rows). A surface-feature predictor flags anchored turns at AUC= 0.71 (+0.07 over a length-only baseline) without additional generation, providing a cheap online signal for CoT monitors calibrated on short conversations. Single-turn CoT benchmarks, by construction, cannot detect this gradient.

Value alignment of Large Language Models (LLMs) has been shown to be culturally biased toward Western norms. This results in the mishandling of local values in multilingual societies such as Sri Lanka that have their unique cultural dynamics. Existing benchmarks overlook Sri Lankan-contextualized values in its official language Sinhala, hindering culturally sensitive evaluation and fine-tuning. To bridge this gap, we propose LKValues, the first survey-grounded resource suite for Sri Lankan value alignment. From a trilingual survey of 205 respondents, blending adapted global frameworks and LLM-elicited local constructs, we derive 40 majority-endorsed societal values. Using these values, we construct LKvaluesIT, a Sinhala-English news-derived instruction corpus containing 150K scenario-based instances, and LKvaluesBench, a value-sensitive evaluation benchmark of 1,000 instances. We evaluate a set of proprietary and open-weight LLMs with LKvaluesBench. We fine-tune three open-weight base models (Qwen3.5-4B-Base, Qwen3.5-9B-Base, and Aya-Expanse-8B-Base). Our experiments show that newer and larger LLMs still exhibit low-resource and cultural value-alignment gaps. LKValues fine-tuning improves Qwen-family models in English and Sinhala, reducing invalid outputs and cross-lingual disparities, though gains remain model-family dependent. These highlight LKValues’ efficacy in embedding Sri Lankan values, offering a replicable pipeline for low-resource, country-specific pluralist value alignment. The datasets are publicly available at https://github.com/NextME14/LKValues.

Large language models routinely produce numerical hallucinations—plausible but incorrect quantities, percentages, and statistics—yet existing benchmarks focus almost exclusively on entity-level factuality. We introduce EcoStatKG, a knowledge graph (KG) of 50,746 statistical triples and 306,231 structural triples from 6,871 Wikipedia articles on environmental sustainability, and NumHallu, a 500-question numerical benchmark grounded in held-out KG topics. We use these resources to diagnose how knowledge-base coverage affects eight retrieval strategies across six LLMs. In the full-index, oracle-coverage condition, simple cosine retrieval reaches 0.989 Numerical Exact Match (NEM), statistically indistinguishable from Graph RAG (0.992; paired-bootstrap p=0.14). In the deliberately constructed held-out condition, which removes answer-bearing triples, retrieval methods fall below the zero-shot baseline (0.506 NEM), exposing the risk that topically related numerical distractors can override parametric knowledge. Similarity-threshold filtering restores KG Cosine to near-baseline performance (0.511), rather than establishing a new state-of-the-art. A bounded clinical-domain proof of concept reproduces the coverage and distractor patterns. The contribution is a diagnostic study and resource: its results clarify when numerical retrieval helps, when missing coverage can make it harmful, and why confidence gating is best viewed as damage control.

Mental health stigma has profoundly harmful impacts but its complexity makes it difficult to evaluate. Stigma may involve explicit derogation, but also subtler forms of blame, fear, paternalistic pity, social distancing, structural exclusion, and discrimination. We introduce a theory-grounded benchmark for automatic evaluation of mental health stigma in online communication, consisting of naturally occurring online news and social media text annotated with a fine-grained taxonomy of stigma across multiple mental health conditions. Our annotation framework comprises a binary stigma-detection task and a multi-level taxonomy covering (i) stigma mode, (ii) domain, and (iii) specific components of certain forms of stigma. We apply this framework to texts mentioning six mental health conditions and evaluate large language models alongside stigma-related classifiers for detecting sentiment, toxicity, and hate speech. Results show that mental health stigma is not well captured by models trained to detect these neighboring constructs, and that LLMs often overpredict stigma unless given explicit operational rules - mirroring the importance of decision rules in human annotation

Expertise in vertical domains requires more than exposing LLMs to vast corpora; it demands the systematic learning of complex logic. Standard fine-tuning often treats data as an unstructured collection, failing to capture the dependencies needed for high-level reasoning. We propose Domain Knowledge Annealing (DKA), a curriculum learning paradigm designed to improve domain proficiency. Inspired by physical annealing, DKA organizes training from local to global: it starts by injecting discrete concepts via entity-centric samples and progressively advances to synthesizing complex, inter-document relationships. This progression allows the model to integrate isolated facts into a coherent knowledge system. Experiments on Linguistics and Law benchmarks show that DKA outperforms standard strategies, achieving state-of-the-art results among comparably sized models in domain-specific reasoning.

Hidden-state probes have been used to predict factual errors in language-model outputs. We study whether the same representations also distinguish four types of prompt-model relationship: known, unknown, ambiguous, and misleading (built on a false premise). On a controlled 200-prompt dataset constructed to balance six surface features, all six pairwise type contrasts are linearly recoverable from hidden states at the position where generation begins, at AUROC ≥ 0.986 on three open-weight instruction-tuned models spanning two families and a fourfold parameter range. The probe exceeds surface-feature, TF-IDF, and frozen sentence-embedding baselines on every pair. Under group-aware evaluation on paraphrased prompts with group-level shuffled controls, probe AUROC remains at least 0.949 on every pair while lexical baselines degrade. These findings frame hallucination as one downstream consequence of a four-type representational structure that also captures ambiguity and false premises.

Language models often appear to become insensitive to how text is tokenized. Models trained with different tokenizers can reach similar language-modeling loss, and prior work shows that they recover word-level information from the raw subword segmentations (a phenomenon referred to as "detokenization"). This raises a natural question: once that information has been recovered, do the resulting hidden states become independent of the tokenizer that produced them? We study this using *tokenizer twins*, pairs of models matched in architecture, training data, and random seed but trained with different tokenizers. We learn small translators between their hidden states at positions where the two tokenizers agree, and then test the same mapping where their tokenization paths differ. Transfer works well at agreement positions, adding less than one nat of loss, but breaks sharply at disagreement positions, where the loss increases by more than seven nats. This gap appears from the first hidden layer and persists through the network. At the same time, word-identity probes can still recover much of the underlying content from these representations. In other words, language models can abstract beyond their individual subword tokens while still carrying a lasting imprint of how the text was tokenized.

Aggregate refusal scores on mixed safe and unsafe prompts can obscure local boundary failures and label-correlated metadata shortcuts. We present PAIRSAFE, Proxy-Resistant Paired Safety Evaluation, a score-agnostic diagnostic protocol built from 150 deterministic, matched XSTest-derived safe/unsafe pairs. PAIRSAFE distinguishes strict pair locks from audited residual fields, then asks whether nuisance proxies collapse, unsafe twins rank above safe twins, and one threshold yields pair-consistent decisions; generated behavior is evaluated separately. Strict source, normalized-type, and pair-ID controls collapse to PSA 0.50 and PCDA 0.00, as required by construction. Residual metadata remain informative: a leakage-safe combined probe obtains nested Cal-PCDA 0.397, yet residualizing model scores against measured metadata leaves substantial paired signal. Direct-logit margins for Qwen2.5-3B-Instruct and SmolLM2-1.7B-Instruct reach PSA/Cal-PCDA of 0.987/0.880 and 0.973/0.760, respectively. Qwen’s secondary generation check is lower again (PCGA 0.700). These disagreements show why local ranking, shared-threshold decisions, and generated responses should not be collapsed into one score. PAIRSAFE is a bounded diagnostic protocol, not a new classifier or deployment certificate.

Large Language Models (LLMs) show impressive reasoning capabilities, yet their understanding of culturally specific social dynamics remains limited. We present KaiToM, a Theory of Mind (ToM) benchmark for Thai cultural contexts, focusing on Kreng-Jai, a uniquely Thai concept of social reluctance and consideration. Extending TactfulToM, we construct 882 conversations (39.2K questions) spanning three Kreng-Jai dimensions: White Lies, Avoidance of Confrontation, and Fear-Based Respect. To disentangle language proficiency from cultural knowledge, the dataset includes Thai scenarios at varying cultural depths, general English scenarios, and English translations of Thai content. We evaluate Thai-specialized (Typhoonv2.1, SEA-LION-v3), open-weight (Gemma-3, DeepSeek-V3.2, Kimi-K2, Qwen), and proprietary models (GPT-4o-mini, GPT-5.1, Gemini-3-Flash). Models achieve only moderate performance, struggle with Fear-Based Respect, and cannot reliably identify specific Kreng-Jai types, a failure shared by Thai-specialized models. Human evaluation shows that native speakers significantly outperform models on culturally loaded tasks. Performance also degrades in deep relative to surface-level cultural contexts, and generalist models often perform better on English translations than on the original Thai, suggesting that models capture surface cues but lack deeper cultural alignment. KaiToM supports the development of culturally aware systems for low-resource languages. Code is available at https://github.com/Augustus2011/KAI-ToM.

Humor judgment is subjective and often depends on ambiguity, expectation violation, and contextual reinterpretation, so the reliability of an LLM judge cannot be assessed from final detection performance alone. Existing LLM-based humor research and LLM-as-a-Judge studies largely focus on final performance, generation quality, or agreement with human evaluation, leaving prompt sensitivity and rebuttal robustness in subjective judgment tasks relatively underexplored. This paper analyzes how prompt design, structured interpretation, and user rebuttals shape LLM-based humor judgment. In zero-shot sentence-only judgments, models with similar F1 scores show substantially different FPR–FNR values, shifting between over-accepting and under-predicting humor depending on prompt formulation. Few-shot examples partially reduce these directional fluctuations, but their effects remain model-dependent. We further propose the Incongruity-Resolution Judgment framework, which compares a humor-supporting interpretation with a counter interpretation before producing a final judgment. Rather than uniformly improving performance, this framework serves as a diagnostic structure for examining how competing interpretations reshape model judgment behavior. Finally, explicit user rebuttals show that high initial performance does not guarantee robustness, highlighting the need to evaluate judgment persistence, directional error patterns, and interaction-level judgment trajectories.

Word-in-Context (WiC) remains challenging for language models, despite recent progress on lexical-semantic tasks. We hypothesise that this difficulty arises not only from comparing two contextual uses of a word, but also from the absence of an explicit sense inventory that specifies the relevant level of semantic granularity. We evaluate open LLMs on WiC and traditional Word Sense Disambiguation (WSD) under similar settings. We find that providing candidate senses, similar to what is done in traditional WSD, improves WiC performance in all settings. In general, explicit sense information helps models make more consistent and targeted judgements. Human evaluation further shows that many apparent WiC errors reflect label ambiguity or mismatches between model and annotator sense boundaries rather than simple failures of lexical understanding. In particular, results show that LLMs overthink the sense distinction often leading to errors based on overly fine-grained distinctions.

Brain tumor MRI diagnosis requires models not only to identify lesions but also to perform fine-grained medical reasoning based on imaging modality, tumor location, and key imaging findings. However, existing medical multimodal datasets often lack structured diagnostic semantics, and closed-ended evaluations may overestimate model ability due to option cues or elimination shortcuts. To address this issue, we introduce MM-NeuroOnco, a large-scale multimodal instruction dataset and benchmark for brain tumor MRI diagnostic reasoning, containing 24,726 MRI slices from 20 public data sources and approximately 200,000 multimodal instructions grounded in imaging evidence. We develop a multi-model collaborative pipeline for semantic completion and quality control, which supplements original tumor labels and segmentation masks with key diagnostic attributes, and further construct MM-NeuroOnco-Bench with a no-LLM-inference policy for ground-truth labels. The benchmark adds a “None of the above” option to closed-ended questions to reduce evaluation bias caused by option shortcuts. Experiments show that existing general-purpose and medical-specialist multimodal large models remain limited on this task, with the strongest baseline achieving only 41.88% accuracy on diagnosis-related questions. Fine-tuning on MM-NeuroOnco yields NeuroOnco-GPT, achieving a 27% absolute accuracy improvement and demonstrating the effectiveness of our dataset in enhancing fine-grained multimodal medical reasoning. Code and dataset documentation are available at https://github.com/gfnnnb/MM-NeuroOnco.

Large language models have made substantial progress in mathematical reasoning. However, benchmark development for multilingual evaluation has lagged behind English in both difficulty and recency. Recently, GSM-Symbolic showed a strong evidence of high variance when models are evaluated on different instantiations of the same question; however, the evaluation was conducted only in English. In this paper, we introduce MGSM-Pro, an extension of MGSM dataset with GSM-Symbolic approach. Our dataset provides five instantiations per MGSM question by varying names, digits and irrelevant context. Evaluations across nine languages reveal that many low-resource languages suffer large performance drops when tested on digit instantiations different from those in the original test set. We further find that models robustness in HRL setting do not necessarily translate to LRL. Moreover, proprietary models, such as Gemini 2.5 Flash and GPT-4.1 are less robust to digit, whereas Gemini 3.0 Pro is more robust. Among open models, GPT-OSS 120B and DeepSeek v3 show stronger robustness. Based on these findings, we recommend evaluating each problem using at least five digit-varying instantiations to obtain a more robust and realistic assessment of math reasoning.

Instruction-following benchmarks remain predominantly English-centric, leaving a critical evaluation gap for the hundreds of millions of Indic language speakers. We introduce IndicIFEval, a benchmark evaluating constrained generation of LLMs across 14 Indic languages using automatically verifiable, rule-based instructions. It comprises around 800 human-verified examples per language spread across two complementary subsets: IndicIFEval-Trans, translated prompts from IFEval carefully localized for Indic contexts, and IndicIFEval-Ground, synthetically generated instructions grounded in native Indic content. We conduct a comprehensive evaluation of major open-weight and proprietary models spanning both reasoning and non-reasoning models. While models maintain strong adherence to formatting constraints, they struggle significantly with lexical and cross-lingual tasks – and despite progress in high-resource languages, instruction-following across the broader Indic family lags significantly behind English. We release IndicIFEval and its evaluation scripts to support progress on multilingual constrained generation.

Recent studies challenge the reliability of cross-modal attention mechanisms as an indicator for visual token pruning in Large Vision-Language Models (LVLMs). In this work, we demonstrate that such negative conclusions can be substantially influenced by an overlooked phenomenon termed task-bound coupling, wherein cross-modal attention patterns become entangled with task-specific prompts. Through controlled configurations that decouple visual inputs from downstream queries, we observe that attention distributions can be relatively consistent across the evaluated settings. Concurrently, we identify an empirical scaling phenomenon which we term Attention Fidelity Decay, where the downstream effectiveness of attention maps exhibits semantic dilution in higher-capacity models or advanced architectures. Motivated by these insights, we introduce TAP-LLaVA, a simple training-free framework that leverages localized task-agnostic attention for efficient, reusable visual token pruning across multi-turn sessions. Our approach achieves competitive downstream performance while substantially lowering computational overhead. This suggests that attention can provide a reliable structural prior at moderate model scales when isolated from textual confounders.

Humor generation remains a challenging problem for language models due to its subjective, creative, and culturally sensitive nature. While large language models (LLMs) demonstrate strong generative capabilities, their deployment is often constrained by computational cost. We propose a lightweight, corrective feedback-driven framework in which a compact generator (GPT-2, 140M parameters) produces punchlines under the guidance of an LLM acting as a critic. Unlike traditional knowledge distillation, our method leverages the LLM as a feedback provider rather than a teacher to be mimicked, enabling the student model to iteratively refine its outputs through corrective supervision. We systematically evaluate multiple LLMs (LLaMA3-8B, Qwen3-8B and Mistral-7B) as guides under different strategies, including frozen and learnable configurations, to study their impact on humor quality and diversity. To support responsible humor generation, we curate a high-quality setup–punchline dataset from Reddit and explicitly filter out harmful content. Experimental results show that corrective feedback-guided training substantially improves humor quality, fluency, and diversity over standard fine-tuning and knowledge distillation baselines, while preserving the efficiency of compact models. We observe that, among the evaluated guides, Mistral-7B consistently provides particularly strong guidance on the curated joke dataset. Our code and resources are available at: https://github.com/Ramakrishna176420/Corrective-Feedback-Driven-Humor-Generation-for-Tiny-Language-Models.

Chain-of-Thought (CoT) prompting has significantly enhanced the reasoning capabilities of Large Language Models (LLMs). However, CoT can also produce fluent but logically flawed reasoning chains that make incorrect answers appear more reliable. In this work, we introduce REMIND (Reliability Enhancement via Metacognitive INtrospection and Debiasing), a framework that enables LLMs to learn from their own failures via episodic memory. Unlike standard error-correction methods that rely on external feedback or extensive fine-tuning, REMIND leverages a self-supervised introspection cycle. We first isolate epistemic failure—instances where CoT decreases model confidence despite yielding incorrect results. The model then performs structured introspection to diagnose the root cause and synthesizes a reusable natural language lesson. These lessons are stored in a vector database and retrieved during future inference steps to correct similar future errors. Experiments on eight diverse reasoning benchmarks spanning mathematical, scientific, multi-hop, truthfulness, and commonsense reasoning demonstrate that REMIND significantly improves reliability by effectively converting past CoT failures into actionable insights for unseen problems. Codes are available in https://github.com/newcodevelop/REMIND.

Reinforcement Learning with Verifiable Rewards (RLVR) has been shown effective in enhancing the visual reflection and reasoning capabilities of Large Multimodal Models (LMMs). However, existing datasets are predominantly derived from either small-scale manual construction or recombination of prior resources, which limits data diversity and coverage, thereby constraining further gains in model performance. To this end, we introduce DeepVision-103K, a comprehensive dataset for RLVR training that covers diverse K12 mathematical topics, extensive knowledge points, and rich visual elements. Models trained on DeepVision achieve strong performance on multimodal mathematical benchmarks, and generalize effectively to general multimodal reasoning tasks. Further analysis reveals enhanced visual perception, reflection and reasoning capabilities in trained models, validating DeepVision’s effectiveness for advancing multimodal reasoning.

Predicting next surgical phase, viewed as a form of surgical procedural intent, with high accuracy is essential for robotic real-time surgical systems. Previous approaches struggle with short-term dynamics, occlusions, and phase persistence; end-to-end large vision-language models (LVLMs) can reason from visual inputs but often fail to exploit structured temporal observations. Thus, we introduce SurgAgents, a multi-agent framework for embodied surgical intent prediction. SurgAgents decomposes prediction into specialized agents: (1) a ResNet-based single-frame classifier agent extracts per-frame phase information; (2) a YOLO-based instrument specialist agent extracts per-frame instrument sets; (3) a history descriptor agent composes these facts into frame-level descriptions; (4) a history summarizer agent compresses long histories into a compact temporal narrative; (5) a phase predictor agent assembles the narrative and the current frame into a formatted LVLM prompt; (6) a retrieval agent with a confidence-gated Retrieval-augmented generation (RAG) module selectively retrieves similar case patterns when prediction confidence is low. On Cataract-1K, SurgAgents attains 46.54% macro-F1 with GPT-4o, outperforming the baseline, with systematic ablation studies showing that each agent materially contributes to performance. Latency measurements demonstrate feasibility for robotic real-time surgical support during surgery procedures.

Large language models (LLMs) are increasingly used as role-playing dialogue agents in long-horizon applications such as tutoring, companionship, and narrative interaction, where users may expect stable persona consistency as well as plausible stage-conditioned change. Existing evaluations emphasize static persona fidelity, temporal factuality, or isolated social skills, but rarely test whether role-playing models produce coherent developmental differentiation across ordered stages. We introduce GROWTHEVAL, a psychology-inspired diagnostic framework for assessing stage-conditioned developmental realism in single-turn, stage-isolated role-playing responses. Using stage-conditioned scenarios and multi-axis scoring, GrowthEval evaluates cross-stage patterns in developmental appropriateness, role-play coherence, and canon fidelity across five socio-emotional dimensions. Across the evaluated LLMs, we observe a model-tiered empirical pattern: higher-capacity models respond more consistently to stage conditioning but can over-mature at early stages, whereas lower-capacity models exhibit flatter and weakly differentiated profiles. Reliability and robustness analyses provide evidence for the stability of these findings. Overall, the evaluated LLMs only partially realize coherent, stage-appropriate developmental change. GROWTHEVAL provides an extensible diagnostic framework for analyzing stage-conditioned behavioral differentiation and associated failure modes in single-turn role-playing responses.

The growth of context window lengths in Large Language Models (LLMs) significantly enhances their long-context capabilities but incurs prohibitive memory costs due to the Key-Value (KV) cache. Although low-rank compression of KV cache is a promising remedy, existing methods face a dilemma: offline approaches depend on external calibration data, whereas online approaches incur substantial compute for full-prompt decomposition and reconstruction. In this paper, we propose **S⁴R**, which builds low-rank subspaces from selectively sampled tokens and computes attention over a sparsely reconstructed KV representation. S⁴R uses prompt-aware initialization to build initial key/value bases from a representative prompt subset, trading off calibration-data dependence against prefilling cost. Because fully reconstructing the cache at every decoding step is prohibitively expensive and hurts throughput, we further adopt sparse reconstruction to retain only informative positions during decoding. Extensive experiments on LongBench and RULER with Llama and Qwen model families show that S⁴R achieves up to 5x KV compression with near full-cache accuracy, combining the efficiency of fixed compression with the adaptability of prompt-dependent methods. Our code is publicly available at https://github.com/Draconis98/S4R

Large language models are used by people with very different expertise levels, but existing work studies what users ask more than who is asking. We introduce QUILT (Query User Interaction Linguistic Taxonomy) , a two-dimensional framework that jointly characterizes LLM conversations by task type and user expertise level. Applying QUILT to 3,000 WildChat dialogues, we construct the first large-scale corpus linking expertise level to task-level interaction patterns in open-ended LLM conversations. Corpus analysis reveals clear expertise-dependent asymmetries: debugging queries are strongly associated with expert users, novice users rarely engage in reasoning-intensive tasks, and expert queries are substantially longer across all task types. These patterns are statistically significant and recoverable by automatic classifiers, indicating that expertise differences are systematically reflected in conversational language. Together, QUILT and the QUILT Corpus provide an empirical foundation for studying user heterogeneity in LLM interaction and for developing expertise-aware response strategies.

Evaluating retrieval-augmented generation (RAG) on custom knowledge bases requires task-specific benchmarks of multi-chunk evidence synthesis, yet most existing generation pipelines apply quality control only after the full question set has been solved. We present VERGE, which moves quality control inside the generation loop: a verification agent targets, per item and without an external solver pass, an operational multi-chunk criterion and a QA-integrity criterion, refining items that fail and discarding those it cannot repair. From 11,052 candidates across six domains, VERGE yields 2,854 verified multiple-choice items targeting aggregation- and comparison-style synthesis over topically coherent evidence. We contribute a method-agnostic validation protocol — exhaustive strict-subset ablation, chunk-removal ablation, format-unconfounded open-ended re-evaluation, and inter-chunk-similarity stratification — and use it to localise where the multi-chunk dependence holds. On a stratified sample (n = 250), 69.8% of items not answerable closed-book cannot be answered from any strict subset of their supporting chunks (strict-necessity violation rate 15.2%). A human annotation study finds the verifier reliable (Cohen’s 𝜅 = 0.903) and VERGE items higher in QA integrity and distractor quality than the closest prior method (p < 0.001).

A language model can fail a syntactic test in two distinct ways: by not encoding the relevant structure, or by encoding it but failing to use it at the output. Behavioral evaluation alone cannot tell these apart. We propose a three-level evaluation framework (behavioral deployment, LM-head readout, and probe recoverability) measured on the same items under the same binary decision. Using a compact trilingual (English, Chinese, German) control-dependency benchmark, we find that probe recoverability exceeds or equals LM-head readout, which in turn exceeds or equals behavioral deployment, across seven models and all three languages in the aggregate. The recoverability surplus is never negative across all 14 (model, task) conditions. The disconnect concentrates in subject-control, where a nearest-noun heuristic gives the wrong answer. The single largest gap (0.653) appears on Qwen3-0.6B Instruct in question answering. The gap persists at Qwen3-14B Instruct. Instruction tuning degrades deployment more than encoding in percentage terms. We rule out option-position bias, late-layer erasure, output-formatting artifacts, and probe-training variance. The pattern is consistent with decoding that favors surface shortcuts, and the behavior-probe gap measures the strength of that preference. Activation patching shows the gap is layer-localized. Under instruction tuning, the LM-head-decoded layer shifts approximately ten layers later than the probe-decoded layer. These findings argue that behavioral evaluation understates what models encode, while probing alone overstates what they deploy.

We audit whether benchmarks for LLM-based recovery of scientific assumptions measure corpus-grounded extraction or pretraining memorization. Using author-defined targets for 10 historical AI breakthroughs (frozen 4-case protocol plus 6 post-hoc stress cases) and a "field-wide paradigmatic assumption" prompt, GPT-4o’s top-10 extractions cover the target in 6/10 cases under a generous any-of-top-K criterion—an upper bound that a 10-rater study corroborates for only ∼0.19 of sampled top-ranked pairs (human–rule 𝜅=0.70, 𝛼=0.634). Controlled stress tests structure the diagnosis: a no-corpus control (field name only) still scores 4/4 on the primary shifts (memorization dominates), while a wrong-corpus control collapses to 0/12 across all 4×3 cross-pairings (corpus can be causally read); a five-LLM canonical sweep (GPT-4o 4/4, Qwen 2/4, Claude 1/4, Llama-3.1 1/4, Mistral 0/4) shows the gap is model-specific rather than closed-vs-open, with format compliance a major but not fully isolable contributor; a Min-K% Prob cross-check actually mis-ranks synthetic non-members as more contaminated than real pre-shift papers, motivating the behavioral framing; and synthetic/niche controls are mixed (causal in 1 of 3 synthetic scenarios, one shows corpus harm). We release an audit kit — five factorable controls (no-corpus, wrong-corpus, pair-level precision, ≥2-LLM, multi-seed; ∼$15) — as a reusable benchmark-validity probe, with all artifacts released.

Multimodal sarcasm detection aims to identify sarcasm in image-text posts by capturing how visual content changes the interpretation of short informal text. Existing methods have advanced the task through multimodal fusion, caption-based cues, and fact/sentiment incongruity modeling, but their evidence use is often controlled inside latent feature channels or reduced discrepancy scores. This makes it difficult to decide which visual information should be selectively weighted for a particular post. We propose CAFE, a caption-anchored feature enrichment and evidence routing framework that decides how to use visual evidence for each sample under ambiguity. CAFE enriches each image with two generated descriptions, one for visible facts and one for the image-text relation, then estimates how useful each description is for the current post and fuses caption-calibrated contrast evidence with direct cross-modal evidence for prediction. Experiments show that CAFE achieves state-of-the-art performance on the primary MMSD2.0 benchmark with 87.81+/-0.33% F1 (mean over 5 seeds). Component ablations on the original MMSD provide an additional evaluation.

Sparse autoencoder (SAE) features are increasingly used to interpret language models, with auto-generated natural-language labels serving as the primary interface for understanding what each feature represents. We ask whether these labels generalize: does a feature labeled for a concept actually track that concept across languages and scripts? Using Serbian digraphia as a controlled testbed—the same language written in both Latin and Cyrillic via deterministic transliteration—we first find that SAE feature sets activated by the same content in different languages, scripts, and wordings share substantial overlap (mean Jaccard 0.39 vs. 0.13 random baseline, peaking at 0.57), suggesting genuine cross-lingual semantic features. We then test whether auto-interpretation labels keep pace. They often do not: features whose labels describe semantic content miss the same meaning in Serbian up to 4× more often than within English, and miss Serbian Cyrillic more than Serbian Latin—two scripts that are deterministic transliterations of each other. The gap grows with network depth, yet the labels give no indication that they fail. These results suggest that auto-interpretation labels reflect a feature’s behavior on the languages and scripts a model has seen most in training, rather than the concept itself.

Using large language models (LLMs) to simulate diverse human populations promises to transform computational social science. However, the accuracy of these simulations remains unclear because current evaluations prioritize predicting an average response over capturing the rich diversity of opinions within real human groups. Here, we develop a diagnostic framework measuring both accuracy and viewpoint retention across 10,000 respondent-question pairs from the World Values Survey, spanning twelve countries and six continents. By evaluating nine LLMs across four alignment paradigms (zero-shot, SFT, DPO, GRPO) against ground-truth human responses, we identify a failure mode we term consensus collapse. While training alignment improves point accuracy, we find that it compresses the range of outputs toward a single demographic stereotype. The highest-performing model (Tulu 3 70B DPO, 57.9% accuracy) retains only 50% of real human variance (σr = 0.50). We demonstrate this collapse is within the models’ underlying parameters rather than decoding procedures; temperature scaling up to T = 1.0 alters fewer than 3.7% of predictions. Furthermore, this diversity loss is inequitable, erasing significantly more variation for Global South populations like Nigeria (σr = 0.11) and Ethiopia (σr = 0.16) compared to Western democracies (σr = 0.70–0.87), with accuracy-rewarded GRPO unable to reverse this effect. While mixing an aligned model with an unaligned prior partially mitigates the collapse (reducing W1 from 0.55 to 0.52), substantial gaps remain (Nigeria recovering from σr = 0.14 to 0.47). Ultimately, we find that modern AI alignment methods trade distributional diversity for point accuracy, challenging existing assumptions for using LLMs as substitutes for human populations. Yet, by exposing the mechanics of this collapse, our work provides insight for future methods designed to value and preserve diverse human viewpoints.

We argue, with systematic empirical evidence, that a large language model’s political ideology is not a fixed point, but a *conditional distribution* ℙ(position∣context) over a real political space. We evaluate nine current LLMs using a unified measurement framework anchored by VAA–CHES projection models, which map responses onto three validated dimensions (*lrgen*, *lrecon*, *galtan*) across six contextual axes. Our findings reveal high sensitivity to context: persuasive framing and under-represented languages displace coordinates by up to 0.57 and 0.52 units, respectively, while chain-of-thought reasoning often amplifies rather than dampens paraphrase instability. Despite this local plasticity, the model cohort occupies a remarkably narrow Overton envelope overall, occupying roughly one-third the spread of major European parties. Supported by a multi-trait multi-method (MTMM) analysis, we conclude that a single point cannot summarize LLM political behavior; it must be characterized as a shape. Our code and data are publicly available.

Prompt design affects the moral judgments and refusal behavior produced by large language models (LLMs), yet controlled comparisons remain fragmented across datasets and models. We introduce ProMoral-Bench, a benchmark-style comparison of 10 prompting paradigms across four LLM families. Using ETHICS, Scruples, WildJailbreak, and our 200-pair ETHICS-Contrast set, we report task-level metrics together with Moral Competence Score (MCS), jailbreak/over-refusal Safety Robustness Score (SRS), and Unified Moral Safety Score (UMSS). UMSS is a study-specific relative summary of MCS and SRS, not an absolute measure of model safety or moral grounding. In this fixed-template evaluation, compact exemplar-guided scaffolds attain stronger competence-robustness trade-offs at lower token cost than the particular multi-stage variants tested, although multi-stage prompting occasionally improves jailbreak refusal. The few-shot demonstrations are non-overlapping but drawn from the evaluation distributions, so their advantage should not be read as out-of-distribution robustness. Code and data are available at https://github.com/rohanthomas334-sketch/ProMoral_Bench.

Large Language Models (LLMs) exhibit remarkable capabilities but remain susceptible to adversarial attacks aimed at eliciting harmful responses. Current defense mechanisms, while partially effective, often compromise the models’ performance on benign tasks. To address this issue, we propose a novel two-stage framework to enhance LLM robustness against adversarial attacks. Initially, we train a universal adversarial prompt designed to align the hidden representations of harmful inputs with benign ones, effectively emulating adversarial attack patterns. Subsequently, we utilize this soft prompt to augment training data, strengthening the model’s capability to reject harmful content. We conduct extensive evaluations to empirically demonstrate that our method outperforms existing methods such as Circuit Breaker (CB) and Deliberative Alignment SFT (DSFT), achieving average defense rate of 96.4% on llama3-8b-instruct model, compared to CB (90.6%) and DSFT (87.9%).

Large Language Models (LLMs) have shown potential in simulating human behaviors and performing theory-of-mind (ToM) reasoning, crucial for complex social interactions. We investigate ToM reasoning’s role in aligning agentic behaviors with human norms in negotiation tasks, using the ultimatum game as our referenced task. We initialized LLM agents with different prosocial beliefs (Greedy, Fair, Selfless) and reasoning methods (chain of thought and ToM reasoning of varying levels), examining their decision-making processes and outcomes across multiple LLMs, including reasoning models like o3-mini and DeepSeek-R1 Distilled Qwen 32B. We perform 2,700 simulations to show that ToM reasoning enhances behavioral alignment with human norms, decision-making consistency, and negotiation outcomes. Consistent with prior findings, reasoning LLMs exhibit limited capability compared to ToM-enhanced LLMs, with different game roles benefiting from different ToM orders.Fair proposers and responders accepting offers were the most consistent with their strategic reasonings, whereas all agents showed strong consistencies with human beliefs when rejecting offers, except when the offer was fair. Human verification further revealed that Llama 3.3 70B produces reasoning most consistent with its actions and beliefs. Our findings advance understanding of ToM’s role in human-AI interaction and cooperative decision-making. The code used for our experiments can be found at https://anonymous.4open.science/r/UltimatumToM.

Large language models rely on prompt context to perform in-context learning, follow formats, and adapt to local evidence. This ability, however, also raises a robustness concern: models may reuse irrelevant tokens from earlier context when answering a later query. Prior work identifies this behavior as contextual entrainment and finds attention heads whose ablation reduces it, arguing that these heads are largely task- or relation-specific. But are the mechanisms behind contextual entrainment truly isolated by relation, or do relation-conditioned searches recover reusable components shared across relations? Using factual-distractor relation-completion prompts across 19 lexical-semantic subtypes, we identify attention heads whose ablation reduces entrainment and test how the resulting head sets transfer recovery ability across exact relations, semantic families, and mixed relation distributions. Same-family transfer exceeds different-family transfer, family residuals weaken once globally shared heads are removed, and subtype-specific searches repeatedly rediscover many global heads. Clean-completion and context-use probes further suggest that this shared intervention can reduce harmful copying while preserving ordinary relation completion and simple in-context abilities. Our findings provide a more precise perspective on contextual entrainment and its mitigation: harmful context following is not best addressed by searching for isolated relation-specific circuits, but by using heterogeneous attack prompts to reveal reusable intervention targets whose effects are shaped by semantic relation structure.

Text simplification aims to modify documents and sentences to make them easier to read and understand while preserving their meanings. Different applications require distinct simplification policies, e.g., lexical-oriented simplification and overall rewriting while trading off detail for simplicity. However, achieving such policy-driven control remains an open challenge. In this work, we introduce a simple yet powerful approach that leverages Large Language Model-as-a-Judge (LLM-Judge) to automatically construct policy-aligned training data, removing the need for costly human annotation or parallel corpora. Our method enables the construction of simplification models that adapt to diverse simplification policies at both the sentence and document levels. Remarkably, small-scale open-source LLMs match or surpass GPT-5, as verified by both automatic metrics and edit analysis. The consistent improvements across model families and sizes demonstrate the robustness of our approach.

Pretrained text embeddings are increasingly used as representational maps, yet high category separability does not imply that their geometry recovers expert-defined structure. We study this problem in mental-health-related language, where symptom relations provide an external reference and online communities introduce strong domain, affective, stylistic, and discourse confounds. Using 28 Reddit communities, we compare pretrained and supervised fine-tuned Qwen3 embedding spaces at two scales (0.6B and 4B). We construct category prototypes, evaluate their representational dissimilarity matrices against an expert symptom matrix with representational similarity analysis, and complement this global test with prototype-based typicality and multi-baseline confound controls. Pretrained embeddings show measurable alignment with expert structure within the mental-health subset; fine-tuning strengthens this alignment most at the finest category level; and larger scale improves both zero-shot alignment and supervision-induced gains. Residual alignment remains substantial after controlling for VAD, LIWC, lexical style, and topic-distribution structure. These results suggest that LLM embeddings can recover expert-relevant category geometry, but this recovery is level-dependent and should be tested against explicit confounds rather than inferred from classification alone.

Large Vision-Language Models (LVLMs) achieve strong multimodal reasoning capabilities, yet are still prone to object hallucination - often generating content that is not grounded by the visual input. Existing attention-based training-free mitigating methods rely on visual attention heads identified by high attention weights or sensitivity to visual token perturbations. However, we show that these identified visual attention heads are not necessarily responsible for visual grounding, and using this as guidance is suboptimal. Motivated by this observation, we propose a novel training-free decoding method that identifies visually sensitive heads by counterfactual region masking. We find that these heads provide a clear signal when visual grounding is required and can therefore be used to mitigate hallucination by guiding the suppression of hallucination heads. Across multiple LVLMs and benchmark image captioning and VQA datasets, our method achieves best or near-best performance in reducing hallucination while preserving generation quality.

Missing and misleading evidence are not the same problem in retrieval-augmented generation (RAG), but prompt-based abstention treats them alike. Small frozen models abstain when context is absent, not when it is misleading. We introduce GRAB-RAG (Graded Abstention Benchmark for Retrieval-Augmented Generation), a paired benchmark that tests the same questions across four context conditions (supportive, degraded, missing, and misleading) in Natural Questions and HotpotQA. In the misleading condition, we edit a gold passage to support a wrong answer and place it among other retrieved passages. We test five abstention policies on three small frozen models (3.8B–8B) across two English QA benchmarks. Models abstain reliably when evidence is missing, but under explicit abstention prompting still answer 41.6% of misleading questions, with 63% of those answers echoing the planted wrong entity. Chain-of-thought provides little additional benefit. A generator-side conflict check cuts the rate to 13.3% but discards many correct answers, while a natural language inference (NLI) verifier recovers that coverage but fails when parametric memory and the misleading passage agree on the same wrong answer. Prompt-based abstention asks whether context is sufficient, not whether it is correct. The two-pass checks are diagnostic baselines, not deployable mitigations.

Adaptive learning systems commonly formulate learning path planning as Exercise-Centric (EC) recommendation, where the next step is inferred from item-level interaction logs. Evaluating goal-oriented guidance additionally requires explicit learner goals and curriculum-scale prerequisites: learners with similar exercise records may need different paths toward their targets. We therefore study Knowledge-Centric (KC) personalized learning path planning, where a planner must reason over learner profiles, mastery states, and prerequisite knowledge structures to decide which textbook, unit, and concept should be studied next. To support this setting, we introduce PersonaPath, a benchmark that pairs 2,000 fine-grained learner personas with a hierarchical knowledge graph of 347 textbooks, 1,751 units, and 4,092 concepts across 77 subjects. We evaluate representative LLMs on PersonaPath. Results show that even the strongest LLM reaches only a 29.5% final pass rate in Basic Education, and that the main bottleneck lies in adaptivity, where no model exceeds 44.7% in tailoring paths to individual learners.

The rapid advancement of Large Language Models (LLMs) has necessitated more robust evaluation methods that go beyond static benchmarks, which are increasingly prone to data saturation and leakage. In this paper, we propose a dynamic benchmarking framework for evaluating multilingual and multicultural capabilities through the social deduction game Spyfall. In our setup, models must engage in strategic dialogue to either identify a secret agent or avoid detection, utilizing culturally relevant locations or local foods. Our results show that our game-based rankings are consistent with Chatbot Arena rankings on the overlapping model. However, we find a significant performance gap in non-English contexts: models are generally less proficient when handling locally specific entities and often struggle with rule-following or strategic integrity in non-English languages. We demonstrate that this game-based approach provides a scalable, leakage-resistant, and culturally nuanced alternative to traditional NLP benchmarks. The dataset (game history) and source code can be accessed here.

In this work, we introduce BAREC-X, the first explanation-annotated Arabic readability dataset, comprising 1,793 sentences drawn from the Balanced Arabic Readability Evaluation Corpus (BAREC) and labeled by expert annotators with linguistic justifications aligned with the BAREC guidelines. Annotators reached an average inter-annotator agreement of 93.3% under the At-Least-1 metric (73.9% Exact Match at the Category+Reason level, and 81.9% at the Category level). We frame readability explanation as a gold-level-conditioned classification task: given a sentence and its gold readability level, a system must return the linguistic reasons that justify the label. We present the first systematic benchmark for this task on Arabic. We evaluate two contrasting paradigms: (i) eight LLMs under zero- and few-shot prompting, and (ii) a fully interpretable, training-free rule-based system grounded in 67 linguistic features spanning morphology, syntax, vocabulary, and syllabic structure. Gemini 3 and GPT-5.5 achieve the highest exact-match agreement on fine-grained reason prediction (55.8% and 56.4% Category+Reason), while the rule-based system achieves the strongest exact-match performance at the Category level (65.9%). Together, BAREC-X and our benchmark establish a transparent foundation for interpretable Arabic readability explanation.

Audio large language models (LLMs) enable unified speech understanding and generation, but adapting them to linguistically complex and dialect-rich settings such as Arabic-English remains challenging. We present a controlled study of multi-task instruction tuning for an Arabic-centric audio LLM across generative tasks including ASR and speech and text summarization, and discriminative tasks including dialect and emotion recognition, in a resource-constrained setting. To support end-to-end Arabic speech summarization, we introduce AraMega-SSum, a first speech summarization resource for training and benchmarking Arabic-centric Audio-LLMs. We compare four training strategies (i) Uniform Mixing (UM), (ii) Task-Progressive Curriculum (TPC), (iii) Aligner-Based Diverse Sampling (ADS) for training-time batch construction, and (iv) A two-stage TPC→ADS strategy. Our results show a clear task-dependent trade-off: TPC and uniform mixing favor generative tasks, while ADS is strongest for SER. The two-stage TPC→ADS strategy achieves the best DID performance across both backbones while preserving competitive generative quality. The adapted models also outperform zero-shot proprietary baselines, including Gemini-2.5-Pro, on discriminative tasks. We made AraMega-SSum and all experimental resources available to the community (https://huggingface.co/datasets/QCRI/AraMega-SSum).

Post-training quantization facilitates efficient LLM deployment, yet it disproportionately erodes reasoning and mathematical performance. To recover fidelity without costly fine-tuning, we identify a critical vulnerability in existing output-space methods: the tendency to overfit the limited calibration manifold, which necessitates structural regularization for unseen data. We propose **ROCOB** (**R**esilient **O**utput **C**ompensation with **O**ptimal **B**udgeting), a training-free framework based on *Dual-Manifold Alignment*. ROCOB couples a Principal Activation Component (PAC), which minimizes error on the calibration manifold, with an Orthogonal Geometric Component (OGC), which recovers residual weight-space structure without fitting a second calibration objective. To compare methods fairly under the same added-parameter budget, ROCOB uses Global Spectral Energy Maximization (GSEM) to allocate ranks according to recovered spectral energy and module cost. Across Qwen3 and Llama-3 models, quantizers, and reasoning benchmarks, ROCOB improves average post-quantization recovery in most evaluated model–quantizer settings while preserving a training-free construction pipeline.

Performing causal reasoning akin to humans is fundamental to achieving strong artificial intelligence. However, existing studies on causal reasoning and large language models (LLMs) often underestimate the role of human causal cognition. Human causal reasoning is far more than recognizing causes and effects. Humans often factor in many other considerations when establishing a causal link between two events, such as norm violations, intentionality, morality, counterfactual scenarios, etc. To simulate such reasoning, it is intuitive to identify cause-effect pairs using token causality formalisms while modeling the factors that influence human causal judgment using findings from cognitive science experiments. To support the evaluation and modeling of human-like causal reasoning in LLMs, we introduce HCR-Bench, a challenging benchmark with 1,093 instances and detailed human causal reasoning step annotations, and HCR-Reasoner, an approach that systematically integrates theories of token causality and cognitive science into LLMs for human-like causal reasoning. Results suggest that HCR-Reasoner consistently improves LLMs’ causal alignment with humans, and that theory-guided reasoning is highly effective.

With the emergence of large language models (LLMs), there is an expectation that they can effectively extract structured information from complex, real-world documents. However, most LLMs generate paragraph-style answers that are chaotic, disorganized, and untraceable. To bridge this gap, we introduce the Arranged and Organized Extraction Benchmark (AOE), a new bilingual benchmark featuring long-form documents (171k tokens on average) designed to systematically evaluate the ability of LLMs to comprehend fragmented information and reconstruct it into an organized table. Unlike conventional text-to-table tasks, which rely on fixed schema and narrow domains, AOE includes 12 carefully designed tasks across four diverse domains, requiring models to generate context-specific schema tailored to varied input queries. In our experiments, we evaluated 12 open-source and proprietary LLMs. Even the most advanced models struggle significantly.

Large language models (LLMs) have advanced argument mining (AM), yet most approaches treat component classification, relation identification, and explanation generation as separate tasks, limiting coherence and explainability. We propose the first explainable end-to-end AM framework grounded in Toulmin’s theory, jointly predicting argument components, argumentative relations, and structured explanations in a single generation step. This unified formulation encourages LLMs to produce internally consistent argument graphs and reasoning traces aligned with their predictions. Further, we introduce five explainability criteria tailored to AM and grounded in Toulmin’s theory: completeness, correctness, faithfulness, soundness, usefulness, and develop a unified evaluation protocol combining structural metrics, neural similarity, LLM-based judgment, human evaluation and qualitative analysis. Extensive evaluation on several LLMs shows that our approach achieves competitive performance while substantially improving explainability, generating Toulmin-based explanations that are complete, grounded and faithful, uncovering reasoning errors not captured by standard AM metrics, and being consistently preferred in usefulness evaluations.

Large language models (LLMs) have recently shown promise as planners across web, spatial, and embodied domains. However, existing agents often handle information seeking, spatial grounding, and action planning as separate capabilities, limiting their ability to solve long-horizon compositional tasks. To address these challenges, we present **WebWorld**, a web-simulated world model for long-horizon reasoning over web evidence, scene observations, and action plans. WebWorld maintains structured intermediate states across heterogeneous modalities and verifies them through schema-guided hierarchical planning. The framework introduces a Mixture-of-Transformers architecture that coordinates website, scene, and motion experts via multi-modal self-attention. To support training and evaluation, we develop WebWorld-CoT-55K, a large-scale dataset that integrates heterogeneous modalities into schema-guided reasoning traces, and **WebWorldBench**, a curated benchmark for evaluating long-horizon compositional tasks. Training combines cold-start supervised fine-tuning with offline reinforcement learning based on structural, semantic, and planning rewards; inference uses test-time scaling to aggregate multiple reasoning trajectories. Experiments on WebWorldBench show that WebWorld improves the overall score by up to 8.4% over strong LLM-agent baselines, with gains in outdoor reasoning, geolocation, and travel planning, while still leaving substantial room for progress toward human-level performance.

Recent diffusion and autoregressive video generators have substantially improved short-form synthesis, yet composing multiple independently generated clips into a coherent audiovisual story remains challenging. Character identities may drift across scenes, narrative events can become inconsistent, and dialogue or sound may not align with the evolving visual context. We present **DramaAgent**, a hierarchical, agentic, and model-agnostic framework for multi-scene long-horizon audiovisual story generation. Rather than modifying the underlying video backbone, DramaAgent introduces a stateful control layer that combines hierarchical story planning, persistent character conditioning, scene-wise generation, and reflection-guided targeted repair. The framework maintains reusable story and character states across scenes, diagnoses failures along identity, semantic, temporal, and audio-visual dimensions, and maps each diagnosed failure to a corresponding localized repair operation. Experiments on 50 narrative scripts across five video-generation backbones show consistent improvements in semantic alignment, character consistency, narrative fidelity, and scene-level audio-visual compatibility. Controlled same-backbone comparisons further show that these gains cannot be explained solely by stronger generators or additional sampling. Our results suggest that persistent state, failure diagnosis, and targeted repair provide a practical control paradigm for coherent multi-scene audiovisual generation.

Lexical evolution is universal across languages, yet the mechanisms driving neologism formation vary substantially from one language to another. In Chinese, this variation is particularly reflected in the productivity of compounding, where morphemes are combined through word-formation relations to create interpretable lexical forms. This morphological productivity implies that the Chinese lexicon can be viewed not only as a set of attested words, but also as a dynamically latent space with potential neologisms to be realized. However, this perspective still remains underexplored in computation, as current approaches are largely reactive, focusing on detecting attested words from corpora. This raises a question about word-formation itself: which morpheme combinations do the rules license beyond those that corpora already attest? Motivated by this, we propose a proactive creation paradigm for Chinese neologism research. We introduce a three-stage pipeline: KG-Based Candidate Generation, Morphological Discarding, and LLM-Based Filtering. We then probe the candidates with three complementary signals: expert judgment, model perplexity, and corpus attestation. The three signals agree: accepted candidates sit close to real words and clearly apart from rejected combinations. The pipeline yields 18,662 candidates with definitions, examples, and formation metadata, released as material for follow-up work, such as lexical gaps and lexical change. Overall, this study substantiates a proactive creation paradigm for Chinese neologism research, and uses it to characterize what Chinese word-formation rules license beyond what dictionaries have already recorded.

Reliable model fingerprints are essential for protecting large language models (LLMs) against unauthorized redistribution and commercial misuse. In black-box deployment, verification is hindered by defensive filtering of suspected fingerprint queries, as well as by downstream model modifications that may weaken embedded ownership evidence. These risks require fingerprints to be robust in both construction and injection. For construction, prior paradigms face an imperceptibility trade-off: natural-language fingerprints may be accidentally activated, whereas garbled fingerprints are statistically exposed and easier to filter. For injection, existing methods struggle to preserve persistent trigger–target behaviors under model modification. We propose an end-to-end injected fingerprinting framework to address these challenges. Code-mixing Fingerprints (CF) use lowest-perplexity code-mixing under a high-complexity constraint to mitigate this two-sided imperceptibility trade-off. Multi-Candidate Editing (MCEdit) constructs structurally redundant, margin-separated trigger–target mappings to enable graceful degradation under model modification. Extensive evaluations on imperceptibility, detectability, and harmlessness demonstrate robust ownership verification with negligible impact on utility.

Automated fact-checking is a crucial task that supports a responsible information ecosystem. While recent research has progressed from text-only to multimodal fact-checking, a prevailing assumption is that incorporating visual evidence universally improves verification accuracy. In this work, we challenge this assumption and show that the indiscriminate use of visual evidence can reduce accuracy. Building on this finding, we propose AMuFC, a modular fact-checking framework that employs two collaborative vision-language models with distinct roles to enable the adaptive use of visual evidence. Experimental results on three datasets, including WebFC, introduced in this study, demonstrate the effectiveness of adaptive visual evidence use in fact-checking.

On-policy distillation (OPD) for long-horizon language agents fails even when the teacher remains sequence-informative: in a failing Qwen3 configuration, the teacher still separates good from bad rollouts (sequence AUROC ≈ 0.75), yet the local token-gradient alignment 𝛼_diag decays from 0.71 to 0.09 and standard OPD recovers only 1.3% of the teacher–student gap. We restrict our claims to two coupled mechanisms, jointly termed CHOP (Counterfactual Hierarchical On-Policy distillation): (i) a per-state reliability gate routing between per-token reverse Kullback–Leibler matching and per-segment sliced-Wasserstein matching, and (ii) a counterfactual total-variation gate activating privileged information (PI) only when the PI changes the teacher’s distribution. The scope is explicit: CHOP targets low-reliability, long-horizon traces where token-local support matching plateaus; in short-horizon or high-reliability regimes Top-K local-support matching (LSM) is already sufficient and CHOP’s segment branch is rarely activated. CHOP recovers 31.7% of the anisotropic-Qwen3 gap (vs. 1.3%/14.5%/24.7% for OPD/Top-K LSM/Li et al. 2026); on SWE-bench Verified under a fixed OpenHands harness, CHOP-deployable prevents the OPD-induced regression (6.5%\!→\!13.2%, matched to off-the-shelf 11.8%, not a significant gain over off-the-shelf) and oracle PI reveals further headroom (14.2%, 95% CI [11.4,17.3]); on AIME 2024 at a 60K-token stress cap (a diagnostic horizon, not a deployment target), CHOP holds 44.9 vs. 42.8 for Top-K LSM (p=0.032), while standard OPD collapses to 28.4. Compute-matched, leakage, and PI-removed-teacher controls attribute the gain to supervision-object routing rather than extra compute or PI contamination.

Existing paradigms for black-box LLM evaluation by independent third parties share a structural weakness: the testing benchmark must reach the model being evaluated, enabling model developers to game future evaluations through logging or retraining. Ideally, rigorous evaluation would grant evaluators direct access to the model so testing can occur without exposing benchmark contents. However, proprietary black-box models cannot be released to external evaluators, creating a fundamental barrier to independent evaluation. To address this gap, we propose the Private Evaluation Protocol (PEP), a surrogate-based framework for independent evaluation of black-box models without sending the private benchmark to the model developer. PEP releases a small public slice of a held-out benchmark, queries the target model only on that slice, and trains a surrogate from the resulting responses, balancing fidelity to the target model against leakage of the private remainder over a tunable revelation budget. Across six benchmarks, composite leakage scores reliably predict behavioral exposure (r=0.788), and fine-tuned surrogates reach 0.85–0.90 agreement on template-heavy datasets while plateauing below 0.73 on reasoning-heavy datasets. The optimal revelation budget is dataset-dependent, and PEP estimates black-box performance with 5.88 percentage-point MAE, showing the feasibility of private surrogate-based evaluation for structured tasks while protecting the privacy of the held-out benchmark.

Large language models (LLMs) face significant challenges in sustaining long-term memory for agentic applications due to limited context windows. To address this limitation, many work has proposed diverse memory mechanisms to support long-term, multi-turn interactions, leveraging different approaches tailored to distinct memory storage objects, such as KV caches. In this survey, we present a unified taxonomy that organizes memory systems for long-context scenarios by decoupling memory abstractions from model-specific inference and training methods. We categorize LLM memory into three primary paradigms: natural language tokens, intermediate representations and parameters. For each paradigm, we organize existing methods by three management stages, including memory construction, update, and query, so that long-context memory mechanisms can be described in a consistent way across system designs, with their implementation choices and constraints made explicit. Finally, we outline key research directions for long-context memory system design.

Large Language Models (LLMs) increasingly operate in multi-turn interactions that require maintaining latent conversational states, not only explicit facts. We study affective state as an implicit state inferred from sparse contextual cues that remain relevant after topic shifts, and ask whether LLMs expose internal attention signals for routing such evidence at inference time. Prior attention interventions rely on labeled in-domain head profiles and externally specified target spans, limiting transfer across affective-evidence distributions. We propose Self-targeted Internal Attention Boosting (SIAB), which localizes sparse affective heads via causal localization (in detail activation patching) on synthetic minimal pairs and boosts, at the answer position, the content tokens these heads already attend to; both edited heads and target tokens are derived from the model’s own behavior, enabling the same head set to be reused across tasks.We evaluate on ECC and GoEmotions across three LLMs and multiple prompting settings. Across settings, SIAB improves prompt-matched baselines by up to +10.3 macro-F1 on ECC and +5.5 on GoEmotions. Combined with PASTA-style external supervision, it achieves up to +13.84 accuracy points and +12.51 macro-F1 points, while mechanistic analyses trace the gains to sparse, emotion-enriched heads and show that layer proximity affects additivity.

Large language models can be updated after training by modifying parameters, conditioning on edit demonstrations, or supplying retrieved evidence. Yet successful factual updates should do more than change one prompt: they should generalize to related questions, transfer across languages, suppress obsolete facts, and preserve unrelated knowledge. This is the problem of measuring cross-lingual ripple effects of factual updates. The existing evaluations on multilingual knowledge injection, however, focus on high-resource languages like English, and it is unknown whether the findings can transfer to Indic languages. In this paper, we present IndicRipple, a controlled evaluation of multilingual factual updating over 500 factual updates across five domains and five languages: English, Hindi, Marathi, Telugu, and Kannada. We compare parametric editing, in-context editing, and retrieval-based correction using ripple-oriented probes for logical generalization, compositionality, subject aliasing, relation specificity, and forgetfulness. Our results show that current factual update mechanisms show limited cross-lingual ripple effects. IKE performs best overall but reaches only 21.07% strict semantic-plus-language accuracy; RAG often recovers the edited fact but fails to answer in the requested language; and ROME/MEMIT are frequently language-compliant without semantically propagating the update. Strict accuracy is consistently higher for English evaluation than for the Indic-language evaluations. These findings show that direct update success overstates progress: multilingual factual updating must be evaluated as stable cross-lingual propagation, not isolated prompt-level overwrite.

Speaking-style captioning (SSC) aims to generate natural language descriptions of how speech is delivered, capturing paralinguistic attributes such as vocal timbre, prosody, and expressivity. Many downstream applications, including conversational AI agents, controllable speech generation, and large-scale audio annotation, require dimension-specific style captions that describe targeted aspects of speech (e.g., speaker traits, emotion, or delivery style), rather than a single undifferentiated description. However, existing work lacks a unified formulation for controllable speaking-style captioning. We address this gap by formulating SSC as an instruction-following audio-language task, where explicit instructions specify the speaking-style dimensions to be described. We introduce StyleInstructCaps, the first standardized benchmark for controllable speaking-style captioning in audio-language models. StyleInstructCaps provides a dataset and evaluation framework that measures metadata groundedness, hallucination, instruction-following ability, generalization to unseen instructions and audio datasets, and speaker-style consistency. Across nine state-of-the-art audio-language models, we find that strong instruction-following ability does not necessarily imply grounded speaking-style understanding, and that SSC-specific instruction-diverse training substantially improves groundedness, controllability, and speaker consistency.

Large Language Models (LLMs) have significantly advanced text generation capabilities, including tasks like summarization, often producing coherent and fluent outputs. Despite these advances, hallucinations remain a critical challenge. While prior studies primarily focus on detecting or mitigating hallucinations, little attention has been paid to where hallucinations occur within long generated outputs. In this work, we investigate where hallucinations occur in LLM-based long response generation, using long document summarization as a key case study. Focusing on the challenging setting of long context-aware long response generation, we find a consistent and concerning phenomenon: hallucinations tend to concentrate disproportionately in the latter parts of the generated long response. To understand this bias, we explore potential contributing factors related to the dynamics of attention and decoding over long sequences. Furthermore, we investigate methods to mitigate this positional hallucination, aiming to improve faithfulness specifically in the concluding segments of long outputs.

Transactional agents must act on underspecified requests. "Book me a flight from Chicago to San Francisco" omits budget, timing, stop tolerance, and airline loyalty, any of which can change the best option by hundreds of dollars. The challenge is identifying, without a user-specific prior, which unstated attribute preference is most likely to change the recommendation.

We introduce ELICITBENCH, a benchmark of 1,387 travel and housing scenarios pairing real search-result pages (SERPs) with transaction-grounded preference profiles. Each scenario partitions attributes into stated and hidden preferences. An ex-post analytical oracle assigns question-value labels based on the normalized reduction in market-conditioned recommendation regret. Agents are evaluated by task success, question count, and net utility under a super-linear patience cost.

ELICITBENCH reveals that plausible questions are often not decision-relevant: on Gemini 2.5 Flash and Claude Sonnet 4.5, unguided agents ask natural follow-up questions with little downstream value. SERP-aware prompting raises net utility from 0.073 to 0.306 and task success from 21.7% to 57.6% on Gemini, and net utility from 0.221 to 0.346 on Sonnet while asking fewer questions. The selection of items is a second bottleneck: with every preference stated, agents still pick the wrong item in about 11% of the scenarios.

We investigate how small open-weight language models make decisions in medical tasks and whether their failures also occur in other settings. We evaluated nine models on wearable fatigue and intensive-care tasks, where they chose among six recommendations, including continued monitoring. We then extended testing to a non-medical tool router and subsets of the Berkeley Function-Calling Leaderboard and JSONSchemaBench. We introduce the Minimum Viable Audit (MVA) to examine which actions models choose, whether their responses contain usable decisions, and whether they copy prompt content. Comparisons across tasks help establish where these behaviors recur. Under the baseline intensive-care prompt, MedGemma chose continued monitoring in all 450 attempts, achieving 33.3% accuracy. On the router, it used all six tools and answered 144 of 150 requests correctly. Meditron-7B produced unusable responses and copied prompt content across intensive-care monitoring and the non-medical evaluations at both four-bit and eight-bit quantization. MedGemma’s results motivate checking the medical instructions. Meditron’s recurring failures warrant checking the model configuration and how outputs are generated. Differences between tasks and mismatches between medical prompts and scoring rules limit conclusions about the causes of these failures. We recommend reporting selected actions, invalid and copied responses, and results for each prompt condition alongside accuracy.

Table reasoning remains challenging for large language models (LLMs), particularly in tasks that require multi-step inference over long and structured tables. Existing approaches predominantly rely on single-direction reasoning, which limits their ability to explore alternative hypotheses across tasks. In this work, we propose CRAFT, a unified Counterfactual Reasoning Framework that reformulates Tabular question answering and fact verification into a general bidirectional verification process. Our method explicitly constructs both declarative statements and their counterfactual variants. Evidence is then extracted from reasoning along both the original and counterfactual paths, and integrated via a weighted mechanism to arrive at the final answer. Experimental results show that our approach consistently surpasses representative baselines on table reasoning datasets such as WikiTQ and TabFact, achieving especially large improvements on complex question answering. Our framework also significantly mitigates performance gaps between different backbone LLMs. This indicates that counterfactual reasoning effectively overcomes the limitations of single-direction inference, guiding LLMs toward more discerning reasoning and establishing a more principled paradigm for structured reasoning tasks.

While Large language models (LLMs) have proved able to address some complex reasoning tasks, we also know that they are highly sensitive to input variation, which can lead to different solution paths and final answers. Answer consistency across input variations can thus be taken as a sign of stronger confidence. Leveraging this aspect, we introduce a framework where models are systematically pushed to diversify solution paths towards a final answer, thereby testing them for answer consistency across multiple input variations. Focusing on math problems, we induce variations in (i) order of shots in prompt, (ii) problem phrasing, and (iii) languages used. Experiments on a wide range of open-source LLMs of various sizes show that reasoning consistency differs by variation dimension, and that by aggregating consistency across dimensions, our framework enhances mathematical reasoning performance on monolingual datasets GSM8K and MATH500, and the multilingual dataset MGSM. We also discuss the different efficacy of the types of variations induced and how they could be further leveraged in future work.

Current evaluation methods for large language models are coarse-grained and decoupled from generation, producing generic explanations that fail to provide actionable feedback for model improvement. We propose CriticGen, a fine-grained, generation-aware evaluation framework that turns evaluation into actionable control for answer improvement. CriticGen first generates sample-specific evaluation dimensions and scoring criteria, then uses the induced rubric to jointly produce a score, a reason, an executable refinement suggestion, and a refined answer. Experiments show that fine-grained evaluation should be both instance-specific and actionable. CriticGen improves human-rated rubric relevance/coverage from 3.33/4.03 to 3.97/4.24. On the score-consensus-filtered RefineData test set, the best CriticGen variant reaches 0.9556/0.9560 Pearson/Spearman agreement with the reference scores and improves reason/suggestion F1 to 0.7554/0.7900. Independent validation provides additional support: on 180 human-scored instances, CriticGen-Qwen2.5-7B obtains higher observed score correlations than GPT-5, with 0.8868/0.8969 Pearson/Spearman versus 0.8542/0.8615. Crucially, its feedback translates into substantial answer improvement, raising accuracy from 45.36% to 91.26% on 5,000 examples with reliable gold answers, with 47.30% of answers corrected from Wrong→Right and only 1.40% degraded from Right→Wrong.

Large language models (LLMs) increasingly produce evaluative judgments, but rating-level human–model agreement does not establish whether they can replace human inputs in a downstream task. Using HEXACO personality items as a case study, we conduct a preregistered three-stage validation of whether LLM-generated social-desirability ratings can serve as inputs to forced-choice scale construction, a psychometric task that requires costly human ratings. In Stage I, three LLMs served as raters of item-level social desirability for sentence- and adjective-format items, with human ratings serving as references. In Stage II, human- and LLM-based ratings were entered into the same constrained optimization procedure. In Stage III, 400 participants completed the resulting forms, and latent trait estimates were compared using graded Thurstonian item response theory. LLM ratings showed strong item-level correspondence with human ratings (sentence: r = 0.942–0.963; adjective: r = 0.979–0.980), but optimized forms showed partial item overlap and sparse exact pair overlap. Despite these design differences, LLM-based forms produced latent trait estimates that closely matched those from human-based forms (mean r = 0.815 for sentence items and 0.768 for adjective items). These results support the conditional use of LLM-generated ratings as scalable construction inputs while showing why downstream validation is necessary.

Model merging is a practical way to combine several fine-tuned models, but merged models can lose performance relative to their task-specific counterparts. Task accuracy shows the degradation, but not how merging changes the model’s internal representations. We study this question through hidden-state distortion, which measures the difference between a merged model’s hidden states and those of the corresponding task-specific reference model on the same examples. Across two model families, six tasks, and three merge methods, we find that the distortion is highly anisotropic: much of the merged-reference change is concentrated in a small number of directions. Removing distortion components along dominant directions often improves accuracy more than removing those along unrelated directions. When we jointly fit a single distortion basis across tasks for each model family, this removal still improves accuracy, suggesting that the tasks share some useful distortion structure. In addition, the distortion score provides a label-free criterion for choosing among candidate task-arithmetic merge scales, matching supervised selection for both model families.

In-context learning (ICL) for multi-hop question answering (MHQA) depends critically on the choice of demonstrations. Yet most existing methods prioritize semantic similarity, even though semantically similar examples may follow different reasoning paths and therefore be poor demonstrations for the target query. We present STRIDE (STructural Reasoning for In-context Demonstration Extraction), a structure-centric framework for multi-hop QA. STRIDE first infers the latent reasoning structure of a query. It then performs positive–contrastive structural routing, selecting structurally compatible demonstrations as positives and semantically plausible but structurally mismatched examples as contrastives. Finally, it constructs structural prompts that expose compact, role-consistent reasoning traces over supporting evidence. Together, it turns demonstration retrieval from semantic matching into structure-aware problem-solving guidance. Empirical results show that STRIDE consistently improves downstream QA performance across settings.

Many artwork questions begin with vision but cannot be answered by visual recognition alone. A model may need to connect an artwork to its creator, depicted subject, collection, or location, and then follow multiple facts to reach the answer. We introduce ArtHopQA, a multimodal multi-hop question answering benchmark for artwork understanding, containing 4,885 image-grounded questions paired with supporting fact chains. Experiments with open and API large vision-language models show a large gap: even the strongest open model reaches only 36.87% accuracy when given the image and title, while gold-path performance reaches 97.77% to 99.80% when the supporting fact chain is provided. Text-only language models exhibit a similar gap, showing that the difficulty is not confined to visual input. Further analysis reveals high-prior geographic guesses, inconsistent answers under image-title conflict, and limited recovery of intermediate entities even when final answers are correct.

Large language models often lose track of persistent instructions and relevant information as multi-turn conversations grow. We study this cumulative contextual decay through three related failure modes: attention pollution, dilution, and drift. We propose **REA** (Role-aware Heuristic Episodic Attention), a context-management framework that assigns different persistence and representation policies to instructions and episodic interactions. Instructional Memory retains identified global constraints in a dedicated prefix. Episodic Memory preserves user inputs and compresses model replies, while heuristic retrieval selects raw text, compressed representations, or omission for each historical turn. On Long-MT-Bench+, REA improves the judge score from 6.32 to 7.36 on a 10-point scale, a 16.5% relative gain over the Vanilla baseline, and reduces average latency by 2.91×. Additional evaluations show aggregate gains on three backbones spanning 1.7B–7B parameters and on Chinese and English role-playing tasks. These results support role-aware context management as a practical approach to maintaining conversational continuity and instruction adherence.

Fact verification is a critical primitive for ensuring the reliability of Large Language Models (LLMs). While frontier LLMs are effective verifiers, their repeated long-context inference incurs substantial costs; meanwhile, existing specialized verifiers face a fundamental dilemma: label-only models suffer from black-box opacity, while free-form rationale generators, whether trained by distillation or explanation-oriented RL objectives, can produce fluent but not structurally checkable explanations. In this work, we propose AtomicCheck a Protocol-Guided Policy Optimization framework that trains specialized verifiers to produce protocol-guided, locally checkable traces, moving beyond opaque label-only predictions and free-form rationales. Rather than cloning teacher traces or optimizing unstructured explanations, AtomicCheck uses RL to make the verification protocol itself part of the policy objective. To preserve semantic integrity during atomic decomposition, we introduce a Contrastive Decomposition Reward that compares decomposed reasoning with a same-backbone holistic counterfactual and penalizes decompositions that reduce verification accuracy. Extensive experiments demonstrate that AtomicCheck outperforms existing specialized verifiers and approaches strong frontier LLMs while producing structured traces that expose local checks behind the final decision.

E-commerce conversations are increasingly multimodal, yet end-to-end response scores cannot localize which dialogue competence failed, and no public benchmark evaluates finegrained multimodal dialogue understanding for Korean commercial agents. We present KoMMERCE, a Korean multimodal fashion-domain benchmark that coordinates nine fine-grained dialogue understanding tasks on the same conversations: sentiment, intent, and strategy classification;image and text knowledge retrieval;dialogue state tracking; span detection; coreference resolution; and the first formulation of cross-modal dialogue relation extraction. The corpus contains 15,293 dyadic role-play dialogues under two-stage quality control. Baselines on six frontier multimodal LLMs reveal task-conditional patterns that should inform deployment choices: no model wins all nine tasks, and context length affects accuracy differently from one task to another.

Large language model (LLM) safety benchmarks often evaluate input risk, response selection, generation, and tool use separately, making comparisons across tasks difficult. We introduce KoEVD (Korean Ethical Violation Dataset), which links five evaluation and analysis tasks through annotated Korean source utterances: input-risk judgment, candidate-response selection, generation safety, response-strategy analysis, and non-executable mock-tool selection. The tasks share source IDs, with task-specific subsets and candidate sampling where required. Built through an approximately 3.5-year human-governed, AI-assisted process, KoEVD contains 13,552 curated user utterances and 71,395 linked response candidates. Across 12 open-weight and commercial models, observed task profiles differ, and aggregate scores obscure variation across severity, explicitness, utterance type, and topic. In matched generation and mock-tool-selection cases, responses classified as safe coexist with unsafe mock-tool choices; pooled unsafe-call rates are 9.73% and 9.31% for the classifier-safe and classifier-unsafe groups, respectively, with framing-specific differences. These diagnostic measurements do not establish real-world safety. KoEVD supports detailed Korean safety analysis through contextual annotations and source-linked comparisons.

Conventional prompting and prompt-based finetuning offer only weak semantic structure, leaving models to derive task-specific reasoning from limited cues. Existing techniques, such as using demonstrations, chain-of-thought prompting, or retrieval augmentation, can improve performance, but often at the cost of longer, more redundant prompts, higher inference overhead, and reliance on external knowledge sources. We present ConceptWeave, a novel concept-centric prompt-based finetuning approach built on adaptive semantic infilling. Instead of depending on external resources, it extracts concise, task-relevant concepts directly from a small portion of downstream data and organizes them into a dynamically evolving hierarchical memory. During fine-tuning, ConceptWeave identifies informative spans in the input through semantic alignment and augments them with corresponding concept representations, enabling fine-grained, localized conditioning of the model. As training progresses, the concepts are continuously refined and structured into a hierarchy that supports efficient retrieval while preserving semantic richness and scalability. We validate ConceptWeave across a range of NLP tasks, including misinformation detection, question answering, sentiment analysis, and hate speech classification, where it consistently outperforms strong baselines and recent state-of-the-art methods.

Large language models (LLMs) are now deployed worldwide, inspiring a surge of benchmarks that measure their multilingual and multicultural abilities. However, these benchmarks prioritize generic language understanding or superficial cultural trivia, leaving the evaluation of grounded tasks—where models must reason within real-world, context-rich scenarios—largely unaddressed. To fill this gap, we present CulturALL, a comprehensive and challenging benchmark to assess LLMs’ multilingual and multicultural competence on grounded tasks. CulturALL is built via a human–AI collaborative framework: expert annotators ensure appropriate difficulty and factual accuracy, while LLMs lighten the manual workload. By incorporating diverse sources, CulturALL ensures comprehensive scenario coverage. Each item is carefully designed to present a high level of difficulty, making CulturALL challenging. CulturALL contains 2,610 samples in 14 languages of 51 regions, distributed across 16 topics to capture the full breadth of grounded tasks. The experiments show that the best LLM achieve 44.48% accuracy on CulturALL, underscoring substantial room for improvement.

As Large Language Models (LLMs) continue to scale, inference efficiency has emerged as a critical bottleneck. Multi-Token Prediction (MTP) offers a promising avenue to accelerate LLM inference by predicting multiple future tokens simultaneously. Wherein, DeepSeek MTP has gradually become an indispensable component in industrial LLMs. However, it still faces two challenges: limited accept rates of MTP heads, and difficulties in jointly training multiple MTP heads. To address these challenges, we propose MTP-D, a simple yet effective self-distillation method that substantially boosts MTP head accept rates while maximally preserving main-head performance. We further introduce a looped extension strategy for MTP-D, enabling effective and economical extension to multiple MTP heads and yielding significant additional inference speedup (up to 4.55×). Beyond these contributions, we conduct systematic experiments across seven benchmarks to explore and validate key insights into distillation strategies and the scalability potential of DeepSeek MTP. Results demonstrate that MTP-D and the grouped extension strategy consistently enhance MTP head performance and inference efficiency, facilitating the practical usage of MTP in LLMs.

Multi-bit watermarking for large language models (LLMs) enables richer provenance information, but existing schemes remain unreliable for short texts because they rely on local or pre-specified allocation strategies. To address this limitation, we propose a Closed-Loop Error Feedback (CLEF) Multi-Bit Watermarking framework. By treating the LLM as a noisy channel, CLEF tracks the global feature state of generated tokens and dynamically measures its deviation from an ideal multidimensional target trajectory. The resulting dynamic error vector is then used to modulate the logits of candidate tokens via an inner-product scoring mechanism, biasing generation toward tokens that compensate for underrepresented payload dimensions. This closed-loop control enables robust multi-bit embedding with a simple, fully linear decoding process. Experiments on short-text generation show that CLEF improves bit extraction accuracy and complete message recovery rate over the evaluated multi-bit watermarking baselines. Furthermore, CLEF achieves higher bit extraction accuracy under different text perturbations while maintaining acceptable generation quality across downstream tasks such as text summarization and machine translation.

Pragmatic reasoning, including inferring intent, manipulation, and hidden meaning in dialogue, is essential for trustworthy language understanding yet remains underexplored in current large language models (LLMs). We introduce HIPO, a new benchmark designed to assess Hierarchical Pragmatic Tactic in multi-turn Conversations. Inspired by linguistic pragmatics, HIPO operationalizes a hierarchical annotation schema for multi-turn conversations and labels each utterance along one dialogue-level goal and three utterance-level dimensions: communicative intention (why), veracity strategy (how), and illocutionary act (what). To ensure reliable supervision, we design a structured generation pipeline to generate high-quality synthetic dialogues. The benchmark comprises over 4,088 benchmarking utterances with 6,350 contextual utterances across 1,131 dialogues drawn from 31 real-world-inspired scenarios. Benchmarking 22 state-of-the-art LLMs reveals a striking gap: while models achieve higher performance at recognizing surface speech acts(83.3% accuracy), they perform poorly on detecting veracity strategies(32.2%) and speaker intentions(45.1%). These results suggest current LLMs remain challenged by the pragmatic inferences operationalized in HIPO, particularly those involving information manipulation and intended effects beyond literal content. We further show that supervised fine-tuning on additional synthetic data generated by our pipeline can improve performance on HIPO, suggesting a practical path for targeted capability improvement within this benchmark setting.

Fake news detection is often treated as a content-centered task, yet social media posts are embedded in user reactions that express skepticism, endorsement, correction, ridicule, and affective responses. This paper asks what makes user reactions useful for fake news detection. We study performance according to different dimensions: text only, emotions in comments, and number of available comments. We study this question across Fakeddit and Weibo20 under both standard and temporal splits (in a temporal split, most recent posts are included in the testing split). Robustness of results is measured across two encoder families (BERT-base, RoBERTa-base) and two English emotion taxonomies (dair-ai/emotion, GoEmotions). Across these configurations, comment-only representations outperform post-only baselines in most evaluation regimes, though this gain depends on dataset, split, and number of available comments. Emotion-fine-tuned comment encoders produce a temporal-shift-specific significant benefit compared to non-temporal splits.

We compare post-only representations, generic comment representations, emotion-fine-tuned comment representations, and post–reaction fusion models, while varying comment availability, showing that a minimum number of comments is necessary to improve detection of fakes. In addition, we show that fine-tuning comment or reaction-encoders on relevant auxiliary tasks such as emotions has a significant impact on the distribution of embeddings of fake and real posts in their representation space.

In light of these findings, we believe that user responses—and their emotional structure for example—should be taken into account when detecting fake news or compiling datasets for this purpose.

Batch prompting is a practical inference strategy for large language models, but its safety implications remain underexplored. We show that the success of batch prompting for utility does not extend to safety: a harmful question that is reliably refused in isolation can elicit a harmful response when embedded in a batch of benign questions. We identify this as a distinct safety failure mode, not reducible to known vulnerabilities such as in-context learning or long-context effects—and analyze its causes from two complementary perspectives: alignment signal weakening and refusal signal dilution. Across widely used open-source and frontier commercial models, batch prompting consistently achieves high attack success rates as a simple black-box attack. We further show that batch-aware preference optimization effectively mitigates the vulnerability. These findings highlight a blind spot in current safety alignment and point to batch-aware alignment as a necessary step toward robust deployment.

Graduate program selection is a high-stakes decision where applicants struggle to identify programs matching their latent academic profiles. We address this as a retrieval problem: given a student profile, rank 531 graduate programs by estimated admission compatibility, i.e., how closely the student’s profile resembles each program’s historical admit pool. Because applicants rarely report rejections, training data consists only of positive labels, making negative supervision unavailable. We propose PURE (Positive-Unlabeled Retrieval Encoder), a positive-unlabeled biencoder with InfoNCE contrastive loss and a gated structural residual that fuses GPA and experience counts with free-text descriptions. With a novel multi-positive MRR protocol that corrects for false negatives, PURE achieves MRR=0.2696, outperforming BM25 (+4.9 points) and GPT-4o-mini (+8.3 points), while remaining statistically indistinguishable from Biencoder + GPT-4o-mini reranking at negligible cost.

Translation conveys meaning across languages, but does it preserve the personality a character conveys? We introduce an evaluation framework for quantifying how well translation systems maintain perceived character personality, the personality impression a reader forms from a character’s dialogue, described along the five Big Five dimensions; we use these dimensions as descriptive labels for the impression, not as a psychometric measurement of the character. Our framework applies personality predictors to both source and translated utterances, measuring cross-lingual consistency via Spearman rank correlation at the speaker level. We study entertainment subtitle translation, where every utterance is produced by a named character, and compare evaluation conditions that include, shuffle, or withhold the character names. We evaluate five MT systems, including dedicated MT APIs (Google Translate, DeepL) and LLMs (GPT-4o, Qwen3- 32B, Aya-expanse-32B), and additionally report human translation as a reference condition on GAME OF THRONES (English→12 target languages, 11 characters) and ONE PIECE (Japanese→English, 15 characters). Under our evaluation protocol, measured personality consistency varies across systems and language pairs, and high BLEU/COMET translation quality does not reliably imply high personality consistency.

Extracting sentence representations from pretrained large language models (LLMs) is a training-free approach that leverages their inherent semantic understanding. Existing methods primarily rely on prompt engineering to distill core semantics, but the resulting representations often retain language-specific features that hinder semantic tasks. Moreover, these approaches fail to exploit the cross-lingual capabilities of LLMs, remaining confined to monolingual settings. To address these limitations, we propose CLSF, a novel method that actively introduces diverse language perturbations when encoding the same sentence and then fuses the perturbed embeddings, effectively reducing language-specific biases and producing a more universal representation. CLSF is plug-and-play and can be integrated with existing prompt-based methods. Experiments on Semantic Textual Similarity (STS), multilingual alignment, and various downstream tasks show that CLSF consistently improves performance across different LLMs.

Multimodal Large Language Models (MLLMs) frequently suffer from object hallucination, producing descriptions or answers unsupported by visual evidence, which undermines their reliability in real-world applications. This paper begins with an intriguing phenomenon: prefacing queries with statements explicitly contradicting image content often yields more accurate responses than direct questioning. We term this the Contradictory Prompting Effect (CPE). Priming inputs with assertions opposing reality triggers an intrinsic self-verification mechanism within MLLMs, where conflicting claims compel the model to prioritize visual evidence over linguistic priors. Inspired by this discovery, we propose Contradiction-Guided Visual Reasoning (CGVR), a novel training-free framework for hallucination mitigation. CGVR systematically generates deliberate contradictory captions alongside standard ones. These captions drive a dual-channel inference process, in which conflicting signals enforce rigorous cross-checking against visual evidence. Our work reveals MLLMs’ latent self-verification capability triggered by contradictions and establishes a paradigm for lightweight, reliable multimodal reasoning. Extensive evaluations on POPE and MME benchmarks demonstrate that our proposed CGVR significantly mitigates hallucinations across diverse MLLMs. Our code is available at https://github.com/dhr1698/CGVR.

We present a Korean Benchmark Suite (KBS) comprising Ko-ARC, Ko-GSM8K, Ko-EQ-Bench, Ko-WinoGrande, Ko-LAMBADA, and Ko-IFEval, covering six capabilities across 9,396 items. The suite combines review, localization, direct Korean construction, and redesign for Korean, avoiding a translation-only approach to benchmark adaptation. We evaluate Korean-developed models, multilingual open models, and closed API models under protocols that account for model access, prompt format, and scoring differences. Results show that Korean benchmark scores depend on task construction, scoring rules, prompting, and model post-training. These findings suggest that Korean evaluation should report both final scores and the settings used to obtain them.

Automated Knowledge Base Construction (AKBC) is a core NLP task, and recent work proposes generating knowledge bases directly from large language models (LLMs), treating the model itself as the knowledge source. However, LLMs natively possess no representation of entities, leading to duplicate entries as well as conflations.

We propose DisamKB, a methodology for constructing disambiguated KBs directly from LLMs. DisamKB incorporates on-the-fly disambiguation of entities, relations and classes, and is meticulously designed to satisfy both scalability and disambiguation accuracy. We analyze the central design decisions and characterize the trade-offs between accuracy, scale, and cost. We execute DisamKB at scale, obtaining a materialized KB containing over 1M disambiguated entities and 38.4M triples. This represents the first million-scale LLM-native KB with explicit internal canonicalization of entities, relations, and classes, a significant departure from prior Wikimedia-centric works. DisamKB is anonymously available at https://gptkb.org/.

Group-relative policy optimization (GRPO, DAPO, ARPO) has become the dominant recipe for RL fine-tuning of reasoning LLMs, yet in the resource-constrained regime most practitioners train in (a single GPU, small batches) it consistently suffers from rapid entropy collapse that kills exploration, together with sparse credit assignment in long rollouts, which is most extreme under multi-turn tool use but already present in long single-turn chain-of-thought. We address both failure modes with JANUSPO, a drop-in GRPO modification that introduces asymmetric PPO clipping 𝜖₊<𝜖_- to tolerate larger downward than upward updates, pairs it with an asymmetric KL schedule 𝛽_pos<𝛽_neg that doubles as a free entropy regulator without any explicit upper bound, and redistributes the terminal reward through a turn-weighted decomposition w_t(n) that generalises naturally from single-turn CoT to multi-turn rollouts. On Qwen2.5-3B-Instruct / GSM8K, JANUSPO improves from 74.8% to 84.3±0.2% (mean±std over 2 seeds) across 14 controlled ablations, and the same recipe lifts 7B from 85.3% to 93.6% with no hyper-parameter retuning. On multi-turn DeepSearch (103 GAIA + 100 HLE), SFT-warmup-then-JANUSPO is the only RL recipe whose paired comparison against SFT survives best@8 outlier verification, achieving strict dominance on all four reward and F1 columns while a matched-init GRPO baseline’s apparent F1 wins collapse to 0%. Beyond this main result, we document a systematic negative finding: SFT warm-up is a necessary precondition for GRPO-style multi-turn RL on format-unaware base models.

Multimodal aspect-based sentiment analysis (MABSA) aims to identify aspect-level sentiments by jointly modeling textual and visual information, which is essential for fine-grained opinion understanding in social media. Existing approaches mainly rely on discriminative classification with complex multimodal fusion, yet lacking explicit sentiment explainability. In this paper, we reformulate MABSA as a generative and explainable task, proposing a unified framework that simultaneously predicts aspect-level sentiment and generates natural language explanations. Based on multimodal large language models (MLLMs), our approach employs a prompt-based generative paradigm, jointly producing sentiment and explanation. To further enhance aspect-oriented reasoning capabilities, we propose a dependency-syntax-guided sentiment cue strategy. This strategy prunes and textualizes the aspect-centered dependency syntax tree, guiding the model to distinguish different sentiment aspects and enhancing its explainability. To enable explainability, we use MLLMs to construct new datasets with sentiment explanations to fine-tune. Experiments show that our approach not only achieves consistent gains in sentiment classification accuracy, but also produces faithful, aspect-grounded explanations.

As ideation agents become more capable, evaluating them poses an increasing challenge. LLM-as-a-judge approaches are brittle and gameable, while human evaluation is expensive and hard to scale. To address this, we introduce HorizonBench, an evaluation methodology leveraging future scientific discoveries. HorizonBench uses published papers as ground truth for ideation quality. Specifically, ideation agents pretraining and context retrieval data are strictly constrained to pre-publication knowledge, and scored on how well their predicted ideas align with subsequently published work. Using HorizonBench, we conduct the first unified comparison of five published ideation agents across multiple frontier model backbones. We verify that our soft recall metric captures meaningful ideation quality aligned to human judgement via a user study with 51 domain experts. We then uncover a surprising finding: a naïve baseline that receives only the research goal as an input consistently matches or outperforms all existing agents that require retrieval or multi-step reasoning, highlighting the importance of the research goal for ideation. Based on this result, we develop OSCAR, an ideation agent that distils retrieved papers into insights and explicitly reinforces the research goal in each step of the pipeline. OSCAR outperforms all evaluated agents on HorizonBench. We publish the code and an online version of HorizonBench.

Creative text is inherently subjective: any opinion can be equally valid, and there is no objective ground truth. The only way to quantify such subjectivity is to model the crowd of opinions that produces it. Large language model (LLM)-as-a-judge has emerged as a scalable substitute for human raters, but whether an LLM can stand in for the human crowd on AI creativity remains unexplored. We develop a decomposition framework (four main components, 11 sub-components) and a three-step rating procedure, applied identically to human raters and to different LLM judging conditions on AI-generated stories. Humans rate AI creativity with the same subjectivity that creativity research documents for human-authored creative work. Across three structural comparisons (disagreement on the same story including ranking; revision from impressionistic to reflective; sub-component weighting of the final rating), no LLM reproduces all three human patterns. We frame the contribution as a methodological framework plus a quantification of where LLM judges currently fall short of the human ratings in our study. We release all stories, ratings, and code at https://github.com/ku-nlp/creativity-evaluation-artifact.

Retrieval-augmented generation (RAG) over knowledge graphs requires retrievers that can effectively capture both graph structure and semantic information. Recent approaches have explored graph neural network (GNN)-based retrievers to model graph topology in multi-hop reasoning tasks. In parallel, graph language models (GLMs) have emerged as a promising paradigm that integrates graph reasoning and the semantic capabilities of language models. In this work, we introduce a GLM-based retriever and investigate the comparative strengths of GLM-based, GNN-based, and traditional vector-search-based retrievers in single- and multi-hop RAG settings, and with a particular focus on transferability to unseen domains. Our findings suggest that finetuned GLM retrievers generalize better out of domain, achieving SOTA on two multi-hop benchmarks. On in-domain multi-hop QA datasets they remain comparable to prior work, with promising scaling as parameters and subgraph coverage increase. GNN-based retrievers achieve higher graph coverage with an efficient training setup, whereas the vector-search baseline excels at single-hop datasets.

The evolution of AI systems toward agentic operation and context-aware retrieval necessitates transforming unstructured text into structured formats like tables, knowledge graphs, and charts. While such conversions enable critical applications from summarization to data mining, current research lacks a comprehensive synthesis of methodologies, datasets, and metrics. This systematic review examines text-to-structure techniques and the encountered challenges, evaluates current datasets and assessment criteria, and outlines potential directions for future research. We also introduce a universal evaluation framework for structured outputs, establishing text-to-structure as foundational infrastructure for next-generation AI systems.

Agents using the Model Context Protocol (MCP) rely on semantic matching to select tools from third-party servers, exposing a semantic supply-chain risk through attacker-controlled metadata and outputs. We introduce A2M (Attraction-to-Manipulation), a two-stage black-box framework for hijacking MCP agents. The Attraction phase optimizes tool metadata to increase invocation probability; the Manipulation phase uses execution traces to refine adversarial tool returns that steer agents toward attacker-desired outcomes. On LiveMCPBench, direct attacks optimized and evaluated on GLM-4.6 achieve a macro-average malicious tool invocation rate of 93.6% across four scenarios, increase weighted token costs to 32.4× the benign baseline under Cognitive Denial of Service, and attain a mean attack success rate of 74.4% across Information Exfiltration, Environment Integrity Compromise, and Reasoning Derailment. Transfer to four other models without re-optimization yields corresponding macro-averages of 63.6%, 2.7×, and 24.5%. These findings motivate stronger tool vetting and runtime isolation in MCP ecosystems. Code is publicly available at https://github.com/Lilaizhen/A2M.

LLM-as-a-Judge is the standard paradigm for NLP evaluation, yet its systemic reliability remains poorly understood despite being widely treated as a deterministic ground truth. We present a comprehensive reliability audit, stress-testing six frontier models across four benchmarks, five prompt formats, two presentation orders, three sampling temperatures, and ten repetitions per condition. Our empirical analysis reveals severe vulnerabilities: verdicts change across identical replications at temperature zero, position-order swaps flip the majority of verdicts on challenging tasks, and the most deterministic judge achieves perfect consistency by trivially repeating incorrect verdicts, agreeing with ground truth only 51% of the time. To formalize these multi-faceted failure modes, we introduce the trustworthy verdict rate (T), a unified metric capturing the joint probability that an evaluation is reproducible, order-invariant, and accurate. Using T , we derive a theoretical upper bound on accuracy imposed by position bias and show that reliability is item-specific rather than model-level. Finally, we demonstrate that shifting from pairwise win-rate to holistic rubric scoring improves trustworthiness more than any single-format prompting intervention, offering an actionable framework for robust NLP evaluation.

Hallucination, the generation of non factual content by AI systems, poses a serious risk in medical contexts, where errors can directly affect patient outcomes. We present MedHal, a novel large-scale dataset specifically designed to assess capabilities and train models on the task of hallucination detection in medical texts. Current hallucination detection methods face significant limitations when applied to specialized domains like medicine, where they can have disastrous consequences. MedHal addresses this issue by incorporating diverse medical text sources and tasks covering both intrinsic and extrinsic hallucinations, and by providing a substantial volume of data samples suitable for training medical hallucination detection models. We demonstrate MedHal’s utility by training and evaluating a baseline medical hallucination detection model, showing improvements over general-purpose hallucination detection approaches. This resource enables more efficient evaluation and training of medical text generation systems while reducing reliance on costly expert review, potentially accelerating the development of medical AI research.

Deploying large language models in production often requires constraining outputs to structured formats such as JSON or XML, and prior work treats the resulting accuracy loss as an inherent "structure tax". We re-examine this claim by evaluating a battery of models, datasets and schemas, measuring task accuracy, confidence calibration, and hidden-state geometry. The tax turns out to depend on schema design rather than on structure per se: reasoning-first field ordering matches or exceeds free-form accuracy, while answer-first ordering causes steep drops, particularly in smaller models. Format sensitivity scales inversely with a task’s own structural constraints, and schemas that preserve reasoning order also improve calibration with CKA showing greater separability between correct and incorrect representations in middle transformer layers. Our findings indicate that properly designed structured formats can match or exceed free-form performance, reframing the critical question from ’whether to structure’ to ’how to structure’ for optimal reasoning preservation.

Large vision-language models (LVLMs) suffer from visual ungroundedness: they can produce a fluent, confident, and even correct response driven entirely by language priors, with the image contributing nothing to the prediction. Existing confidence estimation methods cannot detect this, as they observe model behavior under normal inference with no mechanism to determine whether a prediction was shaped by the image or by text alone. We introduce BICR (Blind-Image Contrastive Ranking), a model-agnostic confidence estimation framework. The LVLM stays entirely frozen; instead, we extract its hidden states twice per sample, once with the real image-question pair and once with the image blacked out while the question is held fixed. A small, separate probe is then trained on the real-image hidden state, regularized by a ranking loss that penalizes higher confidence on the blacked-out view, teaching it to treat visual grounding as a signal of reliability. At inference only the single real-image pass is used, so BICR adds zero cost beyond a standard single-view probe. Evaluated across five modern LVLMs and seven baselines on a benchmark covering visual question answering, object hallucination detection, medical imaging, and financial document understanding, BICR achieves the best cross-LVLM average on both calibration and discrimination simultaneously, with statistically significant discrimination gains robust to cluster-aware analysis, at 4–18× fewer parameters than the strongest probing baseline.

LLM agents accumulate interaction histories that exceed their context windows, motivating extensive research on memory systems. Yet most existing designs are tailored to a single scenario (such as multi-session chat or a specific trajectory format), offering little evidence that they generalize across the heterogeneous trajectories encountered in deployment. We evaluate twelve baselines spanning specialized memory systems, long-context models, and an unindexed search agent across five scenarios: single-turn QA, multi-session chat, agentic-trajectory QA, memory stress tests, and long-horizon agentic tasks. Notably, DCI-Lite, an unindexed baseline that searches raw text via grep and bounded reads, is among the strongest existing baselines. This suggests that memory performance depends partly on granting the agent active control over retrieval rather than coupling a static store to a single retrieval route. We instantiate this diagnostic insight in **AutoMEM**, a strong baseline that exposes both raw-text and indexed memory views through an agent-controlled interface. Across the evaluated systems, AutoMEM achieves the best mean cross-scenario rank.

Large Language Models (LLMs) can memorize and reproduce sensitive, copyrighted, or otherwise undesirable training content, creating privacy, safety, and regulatory concerns. Machine unlearning offers a practical alternative to full retraining, but many existing methods apply broad or fixed parameter updates that can degrade utility and remain brittle under deployment changes such as post-training quantization, where forgotten knowledge may partially re-emerge.

We propose Forgetting Only What Matters via Unlearning Layers (FOM-UL), a layer-level unlearning framework that selects transformer layers using a forget-to-retain significance score. This score identifies layers with high influence on the forget set and low sensitivity to the retain set, allowing FOM-UL to concentrate updates where they are most effective while leaving most of the model unchanged. This targeted update strategy improves the forgetting-utility trade-off and provides an empirical path toward quantization-resilient unlearning by reducing the chance that small, diffuse updates are erased by low-bit rounding.

Across TOFU, KnowUnDo, and MUSE-style evaluations, FOM-UL reduces residual memorization compared with strong GA, NPO, KLD, SURE, ReLearn, and LUNAR-based baselines while preserving retain-set utility close to the vanilla model. Under 8-bit and 4-bit post-training quantization, FOM-UL maintains a favorable forgetting-utility trade-off, combining competitive memorization suppression with stronger retained utility. Overall, FOM-UL provides an efficient unlearning strategy that improves targeted forgetting, utility preservation, and deployment robustness without claiming formal guarantees of erasure.

Grokking, the phenomenon in which neural networks generalize long after fitting their training data, has been studied in supervised settings on many epochs. LLM pre-training instead involves next-token prediction over an unlabeled corpus, with limited data repetition and no explicit train/validation split. To address this, we propose an exposure-based framework that enables the study of grokking-like dynamics during LLM pre-training. We ground our evaluation in BLiMP minimal pairs, which provide controlled grammatical contrasts. For every BLiMP minimal pair, we identify a critical phrase, the smallest continuous span that captures the grammatical contrast and the phenomenon-relevant context. Examples whose critical phrase appears in the pre-training window are assigned to the proxy-train split; the remaining examples are assigned to the proxy-validation split. Across five grammatical phenomena, we observe delayed generalization. Analyzing pre-training checkpoints before and after generalization shows that grammatical concept vectors become more predictive of grammatical acceptability and occupy a higher-dimensional subspace after generalization. We also find that attention from the critical token to the relevant context token is concentrated in a small number of heads.

Improving tool use in large reasoning models (LRMs) typically demands expensive test-time compute, training resources unavailable to API-only practitioners, or expensive iterative rollouts with execution feedback required by training-free context-engineering methods, all incompatible with resource-constrained deployment. We introduce **GEAR**, a training-free framework that extracts compositional rules from tool-use failures via single-pass meta-reasoning and distills them into inference prompts. We hypothesize that combining explicit rules with examples outperforms either alone. Across 78 API domains, **GEAR**+2-shot outperforms 3-shot by 24.9% on Qwen3-30B, validating our complementarity hypothesis. On Qwen3-4B, **GEAR** exceeds LoRA SFT by 24.3% and RAG-3-shot by 38.5%, while **GEAR**+2-shot beats the state-of-the-art context-engineering method ACE by 38.8% at 4.1× lower cost. Remarkably, Qwen3-30B with **GEAR**+2-shot nearly doubles zero-shot accuracy while remaining under 1 s latency, demonstrating rollout-free adaptation for resource-constrained deployment.

Recent advances in large language models (LLMs) have accelerated automated scientific discovery (ASD), but existing ideation tools still struggle to generate hypotheses that are both testable, literature-grounded, and adaptive to prior experimental outcomes. We developed HARPA to address these challenges by following a human-inspired ideation workflow that mines literature trends, explores hypothesis design spaces, and converges on precise, testable hypotheses by pinpointing research gaps and justifying design choices. Human evaluations show that HARPA-generated hypothesis-driven research proposals performs comparably to a strong baseline system, AI-researcher, in novelty and overall quality, while showing a positive trend in feasibility (+0.78, p<0.05, bootstrap) and a significant gain in groundedness (+0.85, p<0.01, bootstrap) on a 10-point Likert scale. When evaluated with the ASD agent CodeScientist, HARPA produced more successful executions (20 vs. 11 out of 40) and fewer failures (16 vs. 21 out of 40) indicating that expert feasibility judgments track execution success. HARPA also learns a reward model that scores hypotheses based on prior experimental outcomes, achieving approx. a 28% absolute gain over HARPA’s untrained baseline scorer.

Foundation-model agents in multi-step, open-ended environments frequently suffer from compounding errors, where early mistakes contaminate long-horizon trajectories. While Multi-Agent Debate (MAD) succeeds in deterministic domains with external validators (CITATION), agents in subjective, non-deterministic tasks such as persuasion experience problem drift and sycophantic conformity. Focusing on subjective argumentation, we identify semantic leakage in standard Retrieval-Augmented Generation (RAG) as a reproducible trigger for these failures: standard RAG prioritizes vocabulary overlap over logical structure.

To mitigate this leakage, we introduce Taxonomic Strategy RAG (TS-RAG), a systems intervention that routes strategy retrieval through a discrete categorical bottleneck to decouple argumentative structure from topical content. Zero-shot, cross-domain evaluations on r/ChangeMyView show that TS-RAG improves the transfer of abstract argumentative logic where standard semantic retrieval degrades (+14.5 win-rate points over an unaugmented persuader for Gemini Flash 3.0, p<0.05). In asymmetric deployments that mix lightweight and frontier models, TS-RAG raises the win rate of a lightweight persuader against a stronger opponent (70.5% → 78.5%, p<0.05) and reduces rounds to consensus when the persuader is already dominant. Finally, we introduce trace-level diagnostics via a turn-by-turn Debate State Representation (DSR), and show that strict stance-holding constraints on the opponent are necessary to prevent evaluation collapse caused by default agentic sycophancy.

Listwise passage reranking often achieves strong ranking performance, but it incurs substantial computational cost. We propose FLASH, an efficient listwise reranking framework based on semantic token compression and single-step listwise scoring. FLASH represents each query–passage pair with a small set of query-aware semantic tokens and jointly scores all candidates in a single decoder step, enabling query-conditioned joint scoring without expensive token-level listwise processing. We further introduce multi-cardinality training, which allows a single model to support different semantic-token budgets at inference time. We also propose FLASH-LITE, a lightweight variant with offline passage encoding for low-cost online inference. Across TREC and BEIR benchmarks, FLASH improves average ranking effectiveness at comparable online latency and slightly lower FLOPs, while FLASH-Lite substantially reduces query-time cost with a modest effectiveness trade-off. FLASH-LITE enables offline passage encoding, substantially reducing online query-time cost while maintaining competitive reranking performance. Code is provided in the supplementary material.

With the telecommunications field embracing zero-touch management alongside novel O-RAN and AI-RAN frameworks, contemporary telecom networks now function as immensely intricate and heavily softwarized codebases. While automated software engineering (ASE) tools and Software Engineering (SWE) Agents hold the potential to alleviate the critical code generation bottleneck in this domain, their ability to navigate and modify specialized, mathematically rigorous wireless stacks like srsRAN 5G remains unverified. General-purpose coding benchmarks fail to capture the stateful logic and strict requirements of telecommunications, leaving a critical evaluation gap. In this paper, we introduce TeleSWEBench, the first commit-driven benchmark specifically designed to measure an agent’s performance in the telecom domain. We mine real developer commits from the srsRAN 5G repository and distill them into structured test cases across three difficulty tiers (Easy, Medium, and Difficult). Our benchmark consists of 734 questions that are accompanied by executable unit tests. To avoid the rigidity of test cases, we further propose a hierarchical LLM-as-a-Judge framework called TeleJudge that scores agent outputs at the file level and aggregates verdicts holistically. This follows an evaluation based on context and semantic similarity in parallel to a standard unit test-based evaluation. Using this benchmark, we evaluate AIDER, OpenHands, and the Claude Code frameworks, powered by state-of-the-art reasoning LLMs, including Qwen, GPT-OSS, Gemma 4, Kimi, GLM, and Qwencoder 2.5. Our two-stage evaluation reveals that models suffer from a lack of both localization accuracy and functional correctness, with the strongest ASE tools achieving up to 25% of shippable changes.

Large Language Model (LLM) personalization holds great promise for tailoring responses by leveraging personal context and history. However, real-world users usually possess sparse interaction histories with limited personal context, such as cold-start users in social platforms and newly registered customers in online E-commerce platforms, compromising the LLM-based personalized generation. To address this challenge, we introduce GraSPer (Graph-based Sparse Personalized Reasoning), a novel framework for enhancing personalized text generation under sparse context. GraSPer first augments user context by predicting items that the user would likely interact with in the future. With reasoning alignment, it then generates texts for these interactions to enrich the augmented context. In the end, it generates personalized outputs conditioned on both the real and synthetic histories, ensuring alignment with user style and preferences. Extensive experiments on three benchmark personalized generation datasets show that GraSPer achieves significant performance gain, substantially improving personalization in sparse user context settings.

For the practical deployment of the Role-Playing Agents with Large Language Models, prior work has mainly focused on suppressing out-of-character behavior relative to explicitly provided character information such as personas. However, as shown in psychological research, out-of-character behavior can arise not only from contradictions with explicit information but also from mismatches with character images implicitly inferred by users. Modeling user-inference-dependent out-of-character behavior is a challenging task, because it is unclear what kinds of inferences users make from existing information. To tackle this problem, we construct a dataset of 480 dialogue scenarios and collect 2,152 descriptions from users familiar with those titles. Using cases that annotators judged inappropriate despite the absence of contradiction with existing settings, we derive a 12-category typology, organized into general, title-specific, and individual-specific groups. We also conduct an exploratory evaluation of category-guided feedback, finding no consistent overall advantage over standard self-refinement but possible benefits for some characters. Our shareable code and dataset are at https://github.com/CyberAgentAILab/User-Inference-OOC.

Discovering student misconceptions is a key challenge in intelligent tutoring systems, where identifying recurring error-producing patterns can support targeted feedback and adaptive practice. In settings without a predefined misconception taxonomy, clustering is a natural approach for discovering such patterns at scale. Yet clustering is only as good as its representation: full student records contain problem wording, numerical context, and solution-specific details that embeddings can conflate with the student’s error, obscuring the misconception behind the wrong answer. We construct an LLM step-error-annotation-first pipeline that generates step-error annotations, embeds them with or without problem and student response text, and clusters the resulting representations. On MalruleLib, a 101-class misconception benchmark, clustering step-error annotations alone improves clustering accuracy by 0.22 and normalized mutual information by 0.05 on average over clustering the problem and student response, consistent with annotations capturing recurring error patterns more effectively. On Eedi and unlabeled MalAlgoQA, annotation-based representations show higher average held-out clustering accuracy and internal cluster separation, respectively, than their unannotated baselines. To help experts interpret these clusters, the pipeline then generates candidate misconception descriptors and coherence scores for human triage. Overall, these findings show that input representation substantially affects clustering results and that step-error annotations often improve clustering quality.

Large language models (LLMs) are strong sequence predictors, but standard context accumulation offers no explicit mechanism for revising the memory carried between predictions. We propose LLM-as-RNN, an inference-time framework for online learning with frozen LLMs through a bounded natural-language memory state. At each timestep, the model predicts from the current observation and its memory, receives feedback, and rewrites the state to correct errors and retain task-relevant patterns without updating model parameters. Implemented as a structured system prompt, this state makes the adaptation process inspectable through human-readable memory traces. We evaluate LLM-as-RNN on sequential prediction benchmarks in healthcare, meteorology, and finance across Llama, Gemma, and GPT-oss model families. Comparisons with static-context and feedback-aligned baselines examine predictive performance, feedback sources, memory budgets, and inference cost. Our results demonstrate that feedback-driven memory rewriting supports online error correction and sequential prediction under a fixed memory budget, with performance varying across tasks and backbones.

Expert writers calibrate their claims with qualifiers such as “likely benign“ in a radiology report, “may constitute“ in a court ruling, and “suggest“ in a scientific paper. We ask whether this calibration survives when Large Language Models (LLMs) summarize and rewrite professional text. We define epistemic erosion as any change in a source’s expressed certainty, including qualified claims that are dropped or introduced without basis. We measure it with the Epistemic Faithfulness Score (EFS), computed by aligning source and output claims one-to-one with rule-based matching and assigning each distortion to one of six types, namely escalation, attenuation, omission, fabrication, scope shift, and severity warp. On clinical notes, mean EFS for 11 open-weight LLMs ranges from 0.415 to 0.493; three of these models are also evaluated on legal and scientific text. Four hosted models (GPT-4o, GPT-5.4, Phi-4, Mistral-Large-3) reach at most 0.483 in any domain. The most frequent detected distortion is omission (66%), in which a qualified claim is dropped entirely; claims that survive tend to shift toward greater certainty. Model scale and medical adaptation bring no consistent gain, seven prompting strategies bring at most modest gains, and ROUGE-L and BERTScore correlate only weakly with EFS (|r| ≈ 0.1).

Verifier-guided best-of-N decoding has two stages: a generator samples candidate chains of thought, and a verifier selects among them. A subgroup disparity may already be present in the generator’s candidates or be amplified during selection. We separate the two stages using adversarial demographic counterfactuals of medical exam questions. On HuatuoGPT-o1-8B, a substantial race accuracy gap is already present in a single sampled response, before verifier selection. Across seven generators, only Gemma-3-27B-it under Skywork-RM shows significant verifier-induced amplification after multiple-testing correction. Even an oracle selector leaves a residual gap on every medical generator analyzed, showing that correct answers are less often available for one subgroup. Counterfactually paired low-rank adaptation (LoRA) of the generator reduces the single-sample gap on two medical models, although on HuatuoGPT-o1-8B the reduction does not persist after self-judge selection. Single-variant controls indicate that the single-sample reduction is not explained by additional training steps alone. A short fairness instruction modestly reduces the single-sample gap, and a direct preference optimization (DPO) variant of the paired training also reduces it and, on HuatuoGPT-o1-8B, keeps race mention near its baseline rate. For the medical generators studied, candidate generation rather than inference-time selection is the primary locus for mitigating this gap.

Diffusion language models (DLMs) decode several tokens per step. However, the grammar-constrained decoders evaluated here, IG-CD and LAVE, verify candidate tokens sequentially within each denoising step and treat every violation as a rejection, a lookahead verification, or a fallback to autoregressive decoding. We observe that a violation is local to the first token the parser cannot consume, so the rest of the parallel step can be kept. We propose Dgrammar, a training-free decoder that repairs that frontier token in place under the same forward-pass logits, paired with a grammar-gated backoff schedule for the unmask batch size, asynchronous mask construction, and a bounded grammar-guided tail. On JSONSchemaBench medium with LLaDA-8B, Dgrammar improves schema validity by 8.6 percentage points over the strongest DLM baseline while reducing mean latency by 5.8x and eliminating timeouts. Validity and latency gains also hold on the other JSB splits and on Dream-7B, and experiments with a SMILES grammar demonstrate cross-grammar transfer.

In recent years, multimodal large language models have achieved substantial progress, yet their performance remains limited in specialized domains such as circuit diagram understanding. A key bottleneck lies in the lack of diverse and large-scale training data tailored to the structural characteristics of circuit diagrams, together with the high cost of expert annotation. To address this challenge, we propose CircuitPile, a systematic data synthesis framework for circuit visual-language modeling. CircuitPile integrates heterogeneous circuit resources from multiple sources and constructs standardized label spaces and multi-granularity visual representations. Based on this framework, we build 738K domain-specific pre-training samples and 100K task-oriented instruction-tuning samples for the pre-training and supervised fine-tuning stages, respectively. Furthermore, to systematically evaluate fundamental visually grounded circuit perception, we introduce CircuitPerception-Bench, a diagnostic benchmark comprising 2,700 multiple-choice samples across three dimensions: component recognition, spatial understanding, and quantitative reasoning. CircuitPerception-Bench focuses on fundamental visually grounded perception and complements external benchmarks such as AMS Benchmark. Experimental results show that multimodal models trained with CircuitPile achieve significant performance gains on the internal benchmark and demonstrate favorable transferability on the external public AMS Benchmark. For example, applying CircuitPile-based SFT to Qwen3-VL-8B improves the scores on CircuitPerception-Bench and AMS Benchmark from 0.5759 and 0.5067 to 0.7703 and 0.5988, respectively.

Linear probes on language-model hidden states can predict output correctness, but it remains unclear whether they capture transferable correctness representations. We study out-of-distribution correctness-probe transfer across seven datasets, four task families, and three open-weight LLMs: Qwen3-8B, Llama-3.1-8B, and Gemma-2-9B, with all findings validated across five random seeds. Within-family transfer is consistently stronger than cross-family transfer (gap 0.075 AUC), with code generation showing the clearest pattern across MBPP-test, MBPP-train, and HumanEval. We further find that probe weight directions are nearly orthogonal across most dataset pairs despite strong predictive transfer, suggesting that probes do not encode a single canonical correctness direction. Comparing probes to logprob confidence baselines, we find probe advantage is concentrated in in-domain and within-family transfer cells. We also identify a layer-selection confound where early-layer shortcuts can appear transferable in-domain but fail cross-source. Overall, correctness probes transfer selectively rather than universally, making them most useful in-domain and within task families.

Large language models (LLMs) exhibit strong performance on self-contained programming tasks. However, they still struggle with repository-level software engineering (SWE), which demands (1) deep codebase navigation with effective context management for accurate localization, and (2) systematic approaches for iterative, test-driven code modification to resolve issues. To address these challenges, we propose SWE-Adept, an LLM-based two-agent framework where a localization agent identifies issue-relevant code locations and a resolution agent implements the corresponding fixes. For issue localization, we introduce agent-directed depth-first search that selectively traverses code dependencies. This minimizes issue-irrelevant content in the agent’s context window and improves localization accuracy. For issue resolution, we employ adaptive planning and structured problem solving. We equip the agent with specialized tools for progress tracking and Git-based version control. These tools interface with a shared working memory that stores code-state checkpoints indexed by execution steps, facilitating precise checkpoint retrieval. This design enables reliable agent-driven version-control operations for systematic issue resolution, including branching to explore alternative solutions and reverting failed edits. Experiments on SWE-Bench Lite and SWE-Bench Pro demonstrate that SWE-Adept consistently outperforms prior approaches in both issue localization and resolution, improving the end-to-end resolve rate by up to 4.3%.

Autoregressive (AR) models have recently shown strong performance in image generation, where a key component is the visual tokenizer (VT) that maps continuous pixels into discrete tokens. The quality of the VT largely determines the upper bound of AR model performance. However, existing discrete VTs significantly lag behind continuous variational autoencoders (VAEs), leading to degraded reconstructions and poor preservation of fine details and text. Current benchmarks primarily evaluate end-to-end generation quality, making it difficult to isolate VT performance. We introduce VTBench, a comprehensive benchmark for systematic evaluation of visual tokenizers across three core tasks: image reconstruction, detail preservation, and text preservation. Evaluating state-of-the-art VTs with multiple metrics, we find that continuous VAEs consistently outperform discrete VTs, especially in retaining spatial structure and semantic details. Discrete VTs often produce distorted reconstructions and fail to preserve fine textures and text. We release VTBench publicly to facilitate future research on robust, general-purpose visual tokenizers.

The rapid proliferation of mobile applications has resulted in a vast volume of user-generated reviews, which often contain critical insights into bugs, feature demands, and overall user satisfaction. Efficiently organizing and interpreting this feedback is essential for developers seeking to enhance app quality and user experience. In this study, we propose a fine-grained, multilabel classification schema for app reviews, structured as a two-level taxonomy. The first level categorizes reviews into three primary classes: Bug Report, Feature Request, and App Satisfaction Feedback. The second level further refines these categories into subtypes addressing specific concerns such as performance, usability and privacy. To automate this classification, we fine-tune several state-of-the-art large language models (LLMs), including LLaMA, Gemma, Mistral, Qwen, on a manually annotated dataset spanning 10 diverse application domains. Among these, LLaMA 3.1 achieved the highest performance, with weighted F1 scores of 0.90 and 0.79 at the first and second taxonomy levels, respectively. To assess scalability, we applied the fine-tuned LLaMA 3.1 model to label an additional 10,000 unlabeled reviews, creating a silver-standard corpus. This taxonomy-driven approach enables more nuanced analysis of user feedback, supporting targeted app improvements and fostering enhanced user satisfaction.

Large language models (LLMs) are increasingly used for medical consultation and health information support, where safety depends not only on medical knowledge but also on robust responses to unclear, inconsistent, or misleading patient input. However, most existing medical LLM evaluations assume idealized and well-posed patient questions, limiting their realism. We study challenging patient behaviors that commonly arise in real medical consultations and complicate safe clinical reasoning. We define four clinically grounded categories of such behaviors: information contradiction, factual inaccuracy, self-diagnosis, and care resistance. For each behavior, we specify concrete failure criteria that capture unsafe responses. Building on four existing medical dialogue datasets, we introduce CPB-Bench (Challenging Patient Behaviors Benchmark), a bilingual (English and Chinese) benchmark of multi-turn dialogues annotated for these behaviors. We find that although models perform well overall, they exhibit consistent behavior-specific failures, especially when handling contradictory or medically implausible patient information. We further evaluate four intervention strategies and find inconsistent improvements, with some interventions introducing unnecessary corrections. We release the dataset and code.

Predictive vision–language learning aims to learn semantic representations by predicting latent targets rather than reconstructing pixels or generating text. We ask whether latent prediction alone induces the global image–text geometry required for zero-shot matching, and how the location of alignment pressure changes this outcome. In a controlled joint-embedding predictive model family with matched training and evaluation protocols, non-contrastive latent prediction retains information recoverable by supervised adapters and token-level probes but does not form a usable zero-shot image–text space under pooled cosine scoring. A target-contrastive design improves matching, but remains weaker than the directly aligned variant. Our results show that latent prediction is underconstrained at the global embedding level in this setting. Direct global alignment supplies the constraint that makes predictive representations usable for zero-shot scoring. With only a modest direct global alignment loss, latent prediction becomes competitive with matched CLIP and SigLIP baselines and improves several language-sensitive diagnostics.

Vision–language models remain susceptible to multimodal jailbreaks and over-refusal because safety hinges on both visual evidence and user intent, while many alignment pipelines supervise only the final response. To address this, we present SaFeR-ToolKit, which formalizes safety decision-making as an inspectable protocol: during offline data synthesis, a planner specifies a persona, a **Perception**→**Reasoning**→**Decision** tool set, and a constrained transition graph, while a responder emits a typed key–value tool trace before the final answer. To learn this protocol, we train a single policy with a three-stage curriculum (SFT→DPO→GRPO), where GRPO directly supervises tool usage beyond answer-level feedback. We contribute (**I**) the first tool-based safety reasoning dataset of 30,654 training entries (SFT 6k, DPO 18.6k, GRPO 6k) plus 1k held-out evaluation, and (**II**) experiments on Qwen2.5-VL showing large gains in Safety/Helpfulness/Reasoning Rigor on 3B (29.39/45.04/4.98 → 84.40/71.13/78.87) and 7B (53.21/52.92/19.26 → 86.34/80.79/85.34), while preserving general capabilities (3B: 58.67 → 59.21; 7B: 66.39 → 66.81). Code: [https://github.com/Duebassx/SafeR_Toolkit](https://github.com/Duebassx/SafeR_Toolkit).

In this work, we investigate whether detailed, dataset-specific annotation guidelines can replace or augment annotated few-shot exemplars for prompt-based entity detection. To this end, we propose an in-context learning framework that conditions the LLM on a guideline document, task instructions, and optional demonstrations, enabling single-pass, primarily zero-shot nested entity detection. We evaluate multiple instructional aspects from the comprehensive ACE 2005 annotation guidelines, using three guideline-compliant datasets as benchmarks and GPT-5 variants for inference. Based on extensive experimentation, we demonstrate that detailed, dataset-specific instructions, especially those governing mention boundaries, consistently and robustly improve performance, even in conjunction with established few-shot methods. Indeed, even under zero-shot constraints, our guideline approach outperforms state-of-the-art few-shot methods on ACE-2005, achieving up to 81.2 percent F1 without relying on existing labeled datasets. Our findings present detailed annotation guidelines as a promising resource for improving instruction-based information extraction.

Document-level Information Extraction (DocIE) aims to produce an output template with the entities, relations, and events of interest occurring in the given document. Standard practices include prompting decoder-only LLMs using greedy decoding to avoid output variability. Rather than treating this variability as a limitation, we show that sampling can produce substantially better solutions than greedy decoding, especially when using reasoning models. We thus propose ThinkTwice, a sampling and selection framework in which the LLM generates multiple candidate templates for a given document, and a selection module chooses the most suitable one. We introduce both an unsupervised method that exploits agreement across generated outputs, and a supervised selection method using reward models trained on labeled DocIE data. To address the scarcity of golden reasoning trajectories for DocIE, we propose a rejection-sampling-based method to generate silver training data that pairs output templates with reasoning traces. Our experiments show the validity of unsupervised and supervised ThinkTwice, consistently outperforming greedy baselines and the supervised state-of-the-art.

Language models (LMs) are commonly trained with Reinforcement Learning with Verifiable Rewards (RLVR) to enhance their reasoning capabilities. However, since RLVR does not explicitly account for calibration during training, it can lead to severe calibration degradation, including overconfidence. Recent calibration-aware training methods for LMs, which incorporate objectives for uncertainty estimation into training, improve calibration but still exhibit overconfidence under distribution shift, while sacrificing reasoning performance. To this end, we propose RL-ARC, a calibration-aware training framework that jointly leverages reasoning confidence and answer confidence. Specifically, RL-ARC leverages reasoning confidence as an auxiliary signal for calibrating answer confidence, applying it as reasoning-guided regularization for correct cases and as an overconfidence penalty for incorrect cases. Comprehensive results across ID and OOD settings show that, beyond improving calibration, RL-ARC enables reasoning models to adaptively estimate confidence based on the given question without substantially sacrificing reasoning performance, thereby highlighting the importance of reasoning confidence for training reliable reasoning models.

While LLMs demonstrate strong capabilities in English Relation Extraction (RE) tasks, a significant gap remains in assessing their RE performance in low-resource languages, due to a lack of standardized benchmarks. Creating native, language-specific annotated RE benchmarks is expensive and requires human expertise, while relying on translation yields suboptimal, inaccurate datasets that introduce English-centric biases. To tackle this, we propose a scalable, language-agnostic methodology to create RE data by denoising naturally occurring yet noisy alignments of knowledge bases and corresponding texts. We define a proper alignment via a Natural Language Inference (NLI) task, where a relation triplet is entailed by an input text. We empirically assess five denoising strategies and demonstrate that multilingual models outperform language-specific models in the few-shot LLM setting, while Hebrew-specific encoders remain competitive under NLI fine-tuning and are strongest in the label-free cross-training setting. We further find that prompting in the native language achieves performance on par with or slightly exceeding English prompts, confirming that the pipeline operates naturally in the target language. The best open-weights model, Gemma-4-31B-it (F1=0.915), matches the best API model, GPT-5.4 (0.921), and reaches F1=0.844 on a fully held-out, dual-annotated test set. Applying the optimized pipeline, we release HeRE: a novel, massive, 1.5-million-instance silver dataset for Hebrew RE, designed to support data-intensive experimentation and model fine-tuning. HeRE bridges a critical gap in standard Hebrew RE evaluation, while providing the NLP community with a reproducible, language-agnostic pipeline for RE dataset construction in other low-resource settings.

The growing prevalence of multi-sensor wearable devices generates vast amounts of long-term, multimodal time-series data, posing significant challenges for context-aware Large Language Models (LLMs). Current LLM-based health analysis methods typically rely on manually curated context, which becomes impractical with increasing data volume and sensor diversity. To overcome these limitations, we introduce Wearable As Graph (WAG), a novel framework that automates context retrieval for LLMs using personalized knowledge graphs. WAG constructs knowledge graphs mapping relationships between wearable modalities and incorporates user-specific data. We develop a data-driven retrieval pipeline that leverages both global (long-term) and local (short-term) relationships within metrics to identify the most relevant nodes for user queries. We evaluate WAG on a benchmark of over 10k data-associated queries created from multiple wearable datasets. WAG consistently delivers the strongest overall performance, while using substantially fewer input tokens than query-window full-context prompting in the diagnostic setting. These gains remain broadly consistent under human evaluation and across a cross-model generator–judge matrix, supporting structured, query-adaptive context retrieval for LLM-driven analysis of wearable data.

Text-to-SQL is increasingly deployed across trust boundaries between data providers and users. Such deployment must balance three competing requirements: policy compliance, answer coverage, and bounded cost. Existing approaches typically decide refusal based on which columns a query mentions and enforce it stochastically. Whether a query is compliant, however, depends not only on which columns appear but on how they are used, and stochastic enforcement cannot deterministically rule out violations. We formalize this requirement as a column-use policy over semantic use: output, filter condition, and aggregation argument. We integrate the policy by aligning each role with grammar productions tracked by the decoder. The resulting system, PCC-SQL, applies a per-token logits mask that deterministically eliminates single-query column-use violations on the supported SQL fragment in a single decoding pass. Across three benchmarks and three open-source models, PCC-SQL achieves 0% Leakage Rate and Coverage up to 88.7% on Spider-CU, while staying within +10% tokens of direct prompting. We additionally assess semantic alignment with execution accuracy.

In this paper, we survey recent advances in meta-reasoning for Large Reasoning Models (LRMs). Although LRMs have made remarkable progress, they face critical limitations including overthinking on simple queries, under-thinking on complex problems, and ineffective exploration lacking strategic direction. We examine how meta-reasoning addresses these challenges by providing the necessary mechanisms to dynamically regulate and optimize the reasoning process, and propose a framework to categorize existing studies into three core components: the monitor which provides self-perception of the reasoning state, the controller which dictates the meta-policy for regulating actions, and the mechanism which executes the algorithmic mapping between them. By systematically reviewing these developments and foundational challenges, we aim to map the current landscape and identify promising directions for creating more adaptive, efficient, and self-regulating reasoning systems.

PLM-powered neural topic models depend on general-domain pre-training, leading to degraded topic interpretability on specialized corpora. The root cause is geometric: domain-specific terms unseen during pre-training collapse into an indistinguishable region of the embedding space, and neither domain-specific re-training, word-level graph enrichment, nor parameter-efficient fine-tuning can restructure this space without inheriting the capacity ceiling of the underlying encoder. Our key insight is that a learnable graph layer operating on token-level PLM embeddings can acquire corpus-specific semantic structure that the frozen encoder lacks, because token-level graphs preserve document-local context that word-level representations discard and joint optimization with the topic objective reshapes embedding geometry directly from target-domain evidence. We instantiate this insight as DARTopic, a domain-agnostic framework that constructs token-level semantic graphs from frozen PLM embeddings and jointly trains a GNN encoder with variational topic inference. Across three benchmarks spanning general, biomedical, and legal domains, DARTopic consistently outperforms strong baselines in topic coherence and document clustering without any encoder fine-tuning, while demonstrating robustness to PLM choice and favorable runtime efficiency over fine-tuning-based alternatives.

Creative collaboration benefits when collaborators can absorb the nuances embedded in others’ subjective expressions. We focus on especially difficult-to-notice nuances, which we call **minor nuances**, and provide initial empirical evidence that supporting their communication can improve creative collaboration outcomes. We then propose a commonsense-generation-based method that identifies minor nuances in subjective expressions and presents them as cues for collaborators. We evaluated the proposed method in a collaborative image-caption revision experiment designed to reflect brainstorming. Compared with a baseline in which collaborators freely explained the nuances embedded in their own subjective expressions, our method significantly improved output-quality ratings from human evaluators. These results provide initial evidence suggesting that explicitly sharing minor nuances supports richer creative expression by helping convey fine-grained aspects of senders’ inner states that are difficult to communicate through ordinary explanations.

Modeling and predicting political opinions remains a challenging task for AI systems. We introduce PoliSpace, a framework that embeds individuals and political questions into a shared latent space, where vector alignment directly reflects agreement. Individuals are represented as learnable vectors, and questions are encoded via a contrastively fine-tuned language model with individual-specific affine transformations to capture subjective interpretation. This design yields an embedding space that is robust to question rephrasing and sensitive to negation. Evaluated on real-world voting data from Swiss national elections, PoliSpace achieves an 89.81% F1 score on seen opinion prediction for rephrased questions, outperforming prior embedding-based approaches (66.17% F1). We further introduce retrieval and confidence-aware thresholding mechanisms that reach up to 100% F1 at reduced coverage, demonstrating how the learned embedding space can be leveraged to improve prediction reliability. Ablation studies confirm the importance of both individual-specific interpretation modeling and negation-aware encoder fine-tuning. The learned individual embedding space also exhibits clear clustering by political party affiliation, despite no party-label supervision during training, capturing the underlying ideological structure of the Swiss political landscape.

Model merging allows extending an expert model’s capabilities by fusing its weights with another expert. However, task interference has been shown to negatively affect mergeability. To address this, prior work has mainly focused on introducing new merging methods. Despite reducing task interference, the effectiveness of these merging methods is only evident post-merging, requiring a full evaluation of the merged model, which for contemporary LLMs can be expensive. In this work, we study the overlooked predictive power of structural and functional properties of merged experts for mergeability. Our experiments with 8 expert LLMs spanning several programming languages, math, and knowledge show that activation L2 distance and activation cosine similarity emerge as the most important properties for predicting effective merging, contrasting with prior findings on vision models where gradient-based metrics dominated. We further show that task vector L2 distance, a cheaply computed structural metric, provides a mechanistic link between mergeability prediction and training dynamics: it explains why learning rate effects on mergeability are domain-dependent in LLMs, contradicting the universal “higher noise improves merging” principle from vision. This enables a practical intervention where mergeability metrics guide learning rate selection during expert training.

Retrieval-Augmented Generation (RAG) mitigates key limitations of Large Language Models (LLMs), such as factual errors, outdated knowledge, and hallucinations, by dynamically retrieving external information. Recent work extends this paradigm through agentic RAG systems, where LLMs act as agents to iteratively plan, retrieve, and reason over complex queries. However, these systems still struggle with challenging multi-hop questions, and their intermediate reasoning capabilities remain underexplored. To address this, we propose RAGCap-Bench, a capability-oriented benchmark for fine-grained evaluation of intermediate tasks in agentic RAG workflows. We analyze outputs from state-of-the-art systems to identify common tasks and the core capabilities required for their execution, then construct a taxonomy of typical LLM errors to design targeted evaluation questions. Experiments show that “slow-thinking” models with stronger RAGCap performance achieve better end-to-end results, underscoring the benchmark’s validity and the importance of enhancing these intermediate capabilities.

Long-term memory significantly enhances language-model agents but introduces a critical failure mode: retrieval surfaces relevant yet stale, deleted, or mutually inconsistent facts. A query about a current address may retrieve the correct address alongside deleted ones. The answer is present, but the memory set is unsafe. We reframe agent memory as a governed object-selection problem and present MemGov, a substrate-agnostic middleware layer between existing retrieval pipelines and agent context. MemGov wraps retrieved evidence as explicit memory objects, detects must-link and cannot-link constraints, and applies deterministic constraint-aware repair, returning either a certified safe memory set or an auditable obstruction certificate rather than silently passing unsafe combinations. Evaluations on controlled governance scenarios and MemoryAgentBench-style end-to-end tasks, deliberately separating retrieval quality from governance decisions, demonstrate that while raw retrieval recovers answer-bearing evidence successfully, it simultaneously returns detected unsafe memory combinations in 47-51% of cases. The same retrieval substrates governed by MemGov reduce unsafe merges to 0% while preserving answer utility within about 2-4 percentage points, with fully auditable reasoning for all repair and refusal decisions. Additional diagnostics assess eligibility, temporal constraints, and paired answer retention.

Text-driven indoor scene generation and editing require an intermediate representation that language models can both produce and revise. Existing LLM-based systems often rely on scene graphs or global constraint lists, which are compact but underspecify local geometry and make instruction-based edits difficult to localize. We frame the task as structured program generation and local program repair, and propose the Hierarchical Descriptive Scene Language (HDSL), an XML/CSS-style domain-specific language in which rooms, regions, objects, and support surfaces form a tree with parent-relative local coordinates. Our pipeline generates HDSL subtrees recursively with bounded verification, grounds non-virtual nodes through multimodal asset retrieval, and applies force-directed layout optimization to repair boundary and collision errors. For editing, Hierarchical Retrieval-Augmented Generation retrieves the relevant subtree, rewrites only that local context, and merges the result back through a deterministic three-way merge. In our reproduced benchmark, HDSL improves average object coverage, text-scene alignment, and generation time over full text-to-scene baselines while remaining competitive with layout-only reproductions on geometry metrics. On thirty paired editing instructions, HRAG reduces token use by 67.6% and runtime by 68.1%, and cuts non-target removals by 87.5% over full-scene rewriting; attribute ablations confirm that local coordinates and virtual/leaf flags are necessary for valid, locality-preserving edits.

Autoregressive language models are usually trained with next-token prediction under a fixed left-to-right target order. This paper studies target-token organization as a training-time design choice across arithmetic reasoning, multi-label text classification, and text generation. Controlled order studies show that the default NTP order can be far from the best-performing tested organization, while fixed hand-crafted orders such as reverse order do not generalize across tasks. To address this, Max(MI(S; t)) is introduced as a deterministic mutual-information-based heuristic that prioritizes information-rich target tokens. For fixed-length outputs, it constructs task-specific prediction orders; for open-vocabulary text generation, the same principle is applied through lightweight prefix augmentation. Experiments with GPT-style models and pretrained Llama/Qwen variants show that Max(MI(S; t)) often improves over standard NTP and simple baselines such as reverse order, TF-IDF, and conditional-probability selection, without changing model architecture or autoregressive decoding.

LLM-as-a-Judge has emerged as a scalable alternative to human evaluation in multilingual cultural reasoning, yet its reliability for culturally grounded judgments remains poorly understood. We present a human-aligned validation study using the CRaFT framework across Arabic, Bengali, and Spanish, evaluating six judge models, including five open-weight models and GPT-4.1, against paired human annotations across two structurally observable dimensions (Answer Consistency and Explanation Consistency) and three culturally interpretive dimensions (Cultural Fluency, Linguistic Adaptation, and Deviation). Our results reveal a consistent reliability divide: automated judges align substantially better with humans on structural dimensions than on culturally interpretive ones requiring sociocultural reasoning. Reliability further varies across languages, judge architectures, and response-generation models, with several failures reflecting systematic calibration limits rather than prompt underspecification alone. These findings suggest that current automated evaluators are best treated as complementary diagnostic tools rather than replacements for human judgment in culturally sensitive multilingual settings.

Large language models (LLMs) are increasingly studied for clinical decision support, and medical question answering provides an effective testbed for evaluating this capability. In particular, medical multiple-choice questions (MCQs) reflect a core challenge in clinical decision-making: plausible options often share substantial overlapping evidence, while only a few decisive factors distinguish the correct option from other incorrect ones. However, existing retrieval-based and reasoning-oriented methods mainly focus on supplying broadly relevant knowledge or eliciting reasoning paths, rather than identifying the compact differential evidence that separates confusing alternatives. In this work, we present a retrieval-grounded differential evidence framework for medical MCQs. Our method extracts option-specific pieces of differential evidence and organizes them into concise evidence prompts for answer selection. This design better aligns external knowledge with differential reasoning while reducing irrelevant context. Experiments show consistent accuracy gains over strong baselines. Ablation studies further show that our framework outperforms both directly using retrieved evidence chunks and closed-source LLM-generated general hints, and that its improvements are not solely explained by answer-proximal guidance, but are consistent with differential evidence serving as a compact aid for reasoning over competing alternatives. Our findings also offer insights for medical reasoning and other decision-making settings with multiple possible choices.

The rapid proliferation of Vision-Language Models (VLMs) is often framed as enabling unified multimodal understanding, but rests on an under-examined assumption: that current VLMs faithfully synthesise multimodal data. We argue they often do not, and that this gap reflects a trustworthiness problem in the dominant Vision Encoder-Projector-LLM paradigm. Rather than extracting grounded knowledge from visual inputs, state-of-the-art models frequently exhibit functional blindness, exploiting strong language priors to bypass severe visual representation bottlenecks. We challenge the conventional methodology of multimodal evaluation, which relies on data ablation or new dataset creation and therefore conflates dataset biases with architectural incapacity. We propose an information-theoretic departure: the *Modality Translation Protocol*, which preserves a sample’s semantic payload while translating its modality rather than ablating it. From this, we derive three metrics—the Toll (*ToS*), Curse (*CoS*), and Fallacy (*FoS*) of Seeing—culminating in the *Semantic Sufficiency Criterion (SSC)*, a single condition that detects, quantifies, and localises multimodal failures to the visual encoder or the projection head. We further hypothesise a *Divergence Law of Multimodal Scaling*: as language backbones grow, the penalty of the visual bottleneck may widen rather than close. Examining the closest-matching existing benchmarks, motivates our central call: the construction, through *Semantic Equivalence Engineering*, of a large, diverse dataset whose modalities are isomorphic by construction. We frame this not as an experiment awaiting compute, but as a mission for the community: to move beyond "multimodal gain" and build the substrate on which faithful multimodal architectures can be designed.

The rapid expansion of scientific literature far outpaces the speed of manual curation, creating a fundamental bottleneck in extracting structured knowledge from scientific texts. To address this, we introduce an agentic, ontology-guided framework for knowledge graph (KG) construction. Our framework utilizes specialized agents for document understanding, information extraction, ontology conformance, and entity linking. First, an LLM extracts factual knowledge in the form of triples with explicit textual evidence from research papers. Then the triplets undergo a multistage verification process to ground the triplets in the ontology. By enforcing schema constraints and requiring evidence in the form of text fragments, we ensure the data is both semantically consistent and traceable. In an agreement study over 1,000 extracted triplets, Gemini-2.5-flash achieves Cohen’s 𝜅 scores of 0.640–0.700 against three human annotators, matching the human–human agreement range of 0.616–0.689. We applied this framework to 2,000 scientific publications and construct a novel, ontology-grounded KG for climate science with 133,990 nodes and 755,308 relations; of the 23,327 triplets that the publications contribute after validation, 98.96% are judged correct. The pipeline code and KG is publicly available.

A central question in LLM reasoning is whether reinforcement learning (RL) instills genuinely new capabilities or merely reshapes how existing knowledge is expressed during inference. Building on the distribution-sharpening hypothesis, which holds that RL reallocates probability mass toward high-reward trajectories already latent in base models, we ask: can we unlock those latent paths without costly RL fine-tuning? We present Decision-Flow Sampling (DF-Sample), a training-free, data-free inference-time framework that constructs a hierarchical reasoning tree, scores terminal nodes for quality, and back-propagates utilities to inform each intermediate branching decision. Unlike conventional sampling strategies that make purely local step-wise choices, DF-Sample performs explicit global trajectory evaluation before committing to a path, recovering high-quality but low-probability reasoning chains that standard decoding overlooks. On GPQA, DF-Sample achieves 45.6% accuracy, surpassing power sampling (38.9%) and GRPO (39.9%), showing that a training-free method can outperform a trained one. Across three models and four benchmarks, DF-Sample consistently outperforms baselines, indicating substantial latent reasoning potential in pretrained base models.

Large language models are increasingly used for Legal Judgment Prediction (LJP), raising concerns about demographic bias inherited from pre-training corpora and historical legal records. We investigate how fairness robustness changes under model adaptation in LJP through a controlled counterfactual evaluation across gender, ethnicity, and intersectional demographic groups. Using 23 demographic variants over encoder-based models and generative LLMs, we evaluate demographic sensitivity, counterfactual instability, and bias propagation under zero-shot, few-shot, and fine-tuned settings. Experiments show that model adaptation substantially affects fairness. Fine-tuning often improves predictive performance while simultaneously increasing demographic sensitivity and propagating biased supervision signals into downstream judicial predictions, whereas few-shot prompting generally produces smaller propagation effects. We further show that legal-domain specialization does not reliably guarantee fairness: legal-domain encoders span both highly robust and highly sensitive behavior after fine-tuning, and a legal-domain LLM is highly sensitive under few-shot prompting yet comparatively robust after fine-tuning under counterfactual perturbation.

Parameter-Efficient Fine-Tuning (PEFT) has become the standard approach for adapting large language models (LLMs). However, despite reducing trainable parameter counts, many PEFT methods remain memory-bound by token-level activation intermediates that scale linearly with sequence length, limiting their practicality for long-context and on-device adaptation. Unlike prior memory-efficient PEFT approaches that preserve token-level adapter activations, LARS (Low-memory Activation-Rank Subspace) reduces the sequence-length scaling of stored adapter activation intermediates from O(BSRL) to O(BRL) by performing adaptation within a sequence-level low-rank subspace. Through pooled feature extraction, low-rank subspace modulations, and residual projection, LARS reduces the overall memory footprint by 23% on GPUs and 47.8% on CPUs relative to LoRA at S = 1024, with relative savings increasing with context length. Experiments across reasoning, understanding, long-context, retrieval, and token-level tasks show that LARS maintains competitive performance relative to LoRA while substantially improving memory efficiency. Beyond GPUs, LARS also reduces average CPU power and energy consumption per optimization step by 22.43% and 22.06%, respectively. These results demonstrate that activation-efficient adaptation provides a practical path toward resource-constrained and on-device fine-tuning of LLMs.

Large language models (LLMs) often answer ambiguous inputs as if they had a single intended meaning. This is problematic because input ambiguity is an upstream source of uncertainty: when a prompt admits multiple valid interpretations, the key question is whether the model should answer directly at all. Existing input uncertainty methods often rely on LLM-generated statements or clarifications, but such transformations may already disambiguate the input before the ambiguity is measured. We propose a label-preserving perturbation framework for input ambiguity detection in which perturbations remain close to the original input while preserving its label. We instantiate this principle with two detectors: a black-box rule-based perturbation library and a white-box importance-guided method using model-internal signals. Experiments across four datasets in open-domain, conversational, and visual QA show that our methods outperform output-side uncertainty baselines and LLM-paraphrase-based perturbation baselines. Our results highlight the importance of detecting ambiguity before attempting to disambiguate it. Our code is available here.

Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for improving the reasoning capabilities of LLMs. However, the role of supervised fine-tuning (SFT) data in enabling effective downstream RL remains underexplored. In this paper, we investigate a key hypothesis: the diversity of reasoning steps in chain-of-thought (CoT) during SFT is a critical enabler of RLVR performance. To test this hypothesis, we introduce EquivAug, a data augmentation framework that uses symbolic equivalence transformations to generate diverse yet provably correct reasoning paths. By converting CoT solutions into formal symbolic representations, applying equivalence transformations, and reconstructing natural-language explanations, EquivAug produces training data that differs in reasoning steps rather than surface-level lexical variation. Experiments on GSM8K and FOLIO across model scales demonstrate that EquivAug consistently achieves the highest reasoning path diversity after SFT and the strongest accuracy after GRPO. Our results establish that CoT diversity during SFT is positively correlated with downstream RLVR performance.

Sparse autoencoder (SAE)-based activation steering is a lightweight approach for modifying the behavior of large language models (LLMs) by adding small vectors to hidden states during inference. However, designing effective steering interventions remains challenging. In this work, we develop an information-theoretic framework for analyzing SAE activations through empirical feature-firing distributions. The central object of the framework is the Jensen–Shannon divergence (JSD) between such firing distributions, which quantifies how far a generated or intervened state deviates from a dataset-induced target distribution. Based on an analytic divergence response for virtually shifted states, we construct steering vectors toward the target firing distribution. The same divergence can also be used to monitor deviation from another empirical distribution, such as a general valid-language distribution. We demonstrate the effectiveness of the steering in a question-mode generation task study. We further use teacher-forced pulse experiments to show that local interventions induce structured downstream JSD responses in later layers, and apply the framework to a safety-related target–contrast response setting for feature analysis.

Visualized text retrieval, which renders text into images and retrieves them with vision-language models, has been proposed to avoid lossy text parsing and compress long contexts. We ask whether it also bypasses a known flaw of text retrieval: position bias, the tendency to overweight a passage’s beginning. Rendering 1D text into a 2D image may change the bias in two respects: its shape, which position is favored, and its magnitude, how strongly performance varies across positions. We test this by rendering the PosIR benchmark as controlled plain text images and comparing eight visual retrievers with four text retrievers on identical instances. We find that (1) position bias persists, with all visual retrievers exhibiting measurable bias (macro PSI 0.061–0.168); (2) its shape is transformed: seven of eight visual retrievers favor the middle of the page in an inverted-U pattern, contrasting with both the primacy bias of text retrievers and the lost-in-the-middle effect of LLMs with long contexts; (3) its magnitude is sensitive to rendering settings, as taller images amplify PSI and can even reverse positional preference, while font size has a similarly large effect; and (4) under matched backbones, visual retrievers exhibit lower position bias than their text counterparts on longer documents, but this advantage vanishes beyond the pixel budget and at document scale. Visualized text retrieval therefore does not eliminate position bias; it transforms a familiar 1D textual bias into a more complex 2D spatial bias.

Large Language Models (LLMs) often struggle to navigate user requests that contain complex value conflicts, as direct scalar rewards are often ill-equipped to handle these scenarios. To address this challenge, we investigate how chain-of-thought (CoT) reasoning can help improve performance in this domain. Geometrically, we show that CoT correlates with further smoothing the model’s loss landscape in its sharpest direction, helping resolve the optimization instability of traditional scalar rewards. In terms of performance, we demonstrate via relevant downstream benchmarks that CoT may increase capability, demonstrating that value conflict-focused CoT has the potential to be an effective mechanism for better moral reasoning. To capitalize on this potential, we create a new value conflict-focused CoT design that correlates with further smoothing the sharpest direction of the loss landscape and further increases moral reasoning performance. This finding shows that explicitly modifying and improving the design of reasoning dynamics offers a promising avenue for advancing pluralistic alignment in LLMs.

Large language models (LLMs) demonstrate strong instruction-following and reasoning abilities, yet struggle with the socially nuanced understanding humans handle naturally. Humans rely on social sense to interpret intentions, emotions, and relationships in context. We investigate whether and to what extent LLMs exhibit similar abilities when reasoning about social interactions. We propose a structured approach that decomposes social interactions into three aspects and prompts LLMs to generate aspect-based summaries capturing complementary signals. We evaluate these summaries and resulting reasoning within a unified framework across three dimensions, analyzing both LLM–human alignment and the quality of LLM-generated social explanations. Our results show that LLMs achieve moderate agreement with humans and with one another as evaluators and capture some socially nuanced information across aspects, but struggle with interaction-level dynamics shaped by situational norms. Across four downstream CSS tasks, aspect summaries improve aggregate prediction performance across three model families, though effects vary by task. Complementary rationale analysis shows that summaries substantially alter expressed reasoning. Finally, we introduce a benchmark dataset with gold-standard annotations to support systematic study of social understanding and situated reasoning in LLMs.

Large Audio Language Models (LALMs) exhibit strong speech understanding ability, yet it remains unclear whether they meaningfully utilize subtle speech-conditioned emotional cues beyond coarse acted emotions. In this work, we investigate whether the evaluated LALMs exhibit measurable sensitivity to fine-grained emotional variation in speech through controlled perturbation experiments, fine-grained emotion elicitation tasks, and representation geometry analysis. Our results show that vocal delivery variations systematically influence both model behavior and hidden representations, indicating that the evaluated LALMs already encode non-trivial speech-conditioned affective information. However, models struggle under emotionally ambiguous settings, particularly for semantically related fine-grained emotions. Representation analysis further suggests that subtle emotional structure is partially preserved despite weak downstream utilization. Overall, our findings suggest that the evaluated LALMs do not completely ignore subtle speech emotion cues, but instead utilize them inconsistently under ambiguous affective conditions

Policy gradient methods are widely used to align large language model outputs with human preferences, but the log-likelihoods they require cannot be directly computed in Masked Diffusion Language Models (MDLMs). MDLM post-training therefore relies on a surrogate objective based on the Evidence Lower Bound (ELBO), and recent work estimates a blockwise variant of this objective via Monte Carlo methods to support blockwise generation. However, existing estimation schemes randomly sample the block index, leading to unstable estimation. In this work, we propose a full-block estimator that evaluates every block exactly once and averages the results. We theoretically show that the proposed estimator remains unbiased while eliminating the between-group variance arising from block selection, thereby always achieving lower variance than the baseline for the same sample size. In experiments on five benchmarks spanning mathematics, planning, and coding, our method achieves the highest accuracy on four benchmarks and outperforms the random block sampling baseline by an average of 12.79 points.

Language model (LM)-powered agents have shown promise in addressing complex embodied tasks by leveraging the commonsense world knowledge of pre-trained models in interactive physical environments. Yet, real-world deployment requires agents to continually adapt to a stream of tasks, where neither large-scale expert supervision nor reliable external feedback is consistently available. In this paper, we present ASC, a framework for efficient continual embodied task adaptation through amortizing self-correction. Specifically, ASC trains the policy under two objectives: a standard CoT reasoning objective distilled from stronger experts, and a reflexive reasoning objective that internalizes self-correction by correcting initial rationales generated by a previously learned policy. This dual optimization allows the agent to adapt more efficiently through forward knowledge transfer over time, ultimately fostering robust CoT reasoning at inference without requiring explicit reflection. Furthermore, we introduce rationale-wise test-time scaling that dynamically adjusts the depth of CoT reasoning based on the model’s confidence in actions inferred from its own rationales. Through experiments on VirtualHome and ALFWorld, we demonstrate performance superiority over other LM-based planning and continual task adaptation approaches, while achieving strong efficiency in computation, data usage.

Machine-generated text detection remains difficult in Chinese, where systematic benchmarks and large paired corpora are still scarce. We revisit this problem with a compact 4-layer character-level Transformer, trained on a 363.4-million-character Chinese detection corpus, to our knowledge the largest AI-human paired Chinese resource for this task. The resulting data-centric pipeline attains strong scores on Chinese benchmarks and, for the independently trained English variant, state-of-the-art RAID leaderboard numbers that remain stable under 11 RAID adversarial attack types. Ablations expose a double-edged sword role for punctuation: it eases optimization yet encourages overfitting to surface cues; symbol cleaning shifts the model toward deeper syntactic and discourse patterns and improves generalization. Minimal architecture plus rich supervision thus yields compact detectors suitable as a document-level authorship authenticity screen before downstream Chinese text processing; on a laptop RTX 3070 Ti we measure 5.90 x 10^5 tokens/s at under 100 MB VRAM. We present GiantMelon-ZH (Chinese) and GiantMelon-EN (English for RAID): same backbone, disjoint training corpora, transparent data descriptions.

Simulated-patient benchmarks are widely used to evaluate clinical AI agents, yet the patient simulator itself can alter the evidence available to the agent and therefore the conclusions drawn from the benchmark. We introduce PULSE-CARE, a measurement-validity audit for simulated-patient clinical-agent evaluation that quantifies sensitivity to patient simulator behavior, agent architecture, simulator backbone, scorer choice, and demographic framing. Across 24 synthetic longitudinal cases, four dynamic simulator variants, three agent models, and two architectures, the mean per-case score range across simulators is 0.25 (95% CI: 0.21-0.29), and escalation decisions differ across simulator conditions for 60% of agent-case pairs. A 392-trajectory repeated-sampling analysis shows that simulator effects persist after pooling repeated draws, while also revealing substantial within-policy stochastic variation. Nominal model rank reversals occur in 17% of comparisons but disappear under a 0.05 practical-difference margin. Demographic counterfactuals change escalation in 26% of matched pairs, essentially the same rate as the same-input repeated-sampling control, providing no evidence of a demographic effect in this audit. PULSE-CARE combines simulator-sensitivity metrics, EFI composite indices, Patient Simulator Cards, and an Audit Report template to make the measurement conditions behind simulated-patient benchmark results explicit. Our results show that clinical-agent performance reported from a single patient simulator should be interpreted as conditional on the simulator and evaluation pipeline used to obtain it.

Supervised fine-tuning (SFT) is typically viewed as behavior cloning on expert demonstrations. Recent methods recover richer supervision from demonstrations through self-play preference synthesis or bilevel inverse reinforcement learning, yet these require additional optimization stages or separately trained reward models. Motivated by the connection between SFT and inverse soft-Q learning, we propose Dense-Path REINFORCE (DPR), a lightweight post-SFT policy-improvement method that extracts a dense reward signal directly from the SFT training trajectory. We construct a token-level reward from the log-probability gap between the final SFT policy and an intermediate checkpoint, capturing the incremental likelihood gain acquired during training while mitigating the length and early-EOS biases inherent in raw log-probability rewards. DPR reuses the same demonstrations and prompts as SFT, requires no preference data, no external teacher, and no separately trained reward model. Across four pretrained backbones and four instruction-following evaluations, DPR obtains 55.2-66.8% win rates over SFT, improves MT-Bench by 0.22-0.46 points, and matches or exceeds the best demonstration-only baseline in 15 of 16 settings.

Chain-of-thought traces drive strong LLM accuracy but dominate inference cost: output length sets both monetary cost and decoding latency. We ask which tokens in a reasoning trace actually carry the answer. A token-level analysis of the model’s own next-token distribution identifies three redundancies - discourse-element removal, tokenizer-aware abbreviation, and substitution of prose causal connectives with an arrow "X -> Y". All three leave the predictive distribution close to the original under Jensen-Shannon divergence and preserve the greedy next token in the vast majority of prefixes. We instantiate the analysis as CONCISON, a single training-free system prompt that operationalises the three rules and protects numeric, equation, and code tokens verbatim. Across 18 models from six families (2B-120B) and five reasoning benchmarks, CONCISON cuts output tokens by 24-62% on strong models at -1.1pp average accuracy, outperforms token-budget prompting in a single decoding call, and matches trained routers on the accuracy-cost frontier without any training, model pool, or extra infrastructure.

Machine unlearning aims to remove the influence of specific data from trained language models. In real-world deployments, unlearning requests often arrive sequentially, which challenges existing fine-tuning-based methods: fine-tuning each request is costly, accumulates utility loss, and may cause cross-request interference. To address these issues, we propose ICCU (In-Context Continual Unlearning), an in-context continual unlearning framework that induces readable refusal rules from unlearning datasets and applies them at inference time either as a filter or via the system prompt, without modifying model parameters. Because rules are accumulated as an order-independent union, ICCU is compositional and free of cross-request interference, and the original forget-set data can be discarded after rule induction. Extensive experiments show that ICCU effectively suppresses target knowledge while preserving utility, scales across long sequences of requests, and remains robust to paraphrased, cross-lingual, and adversarially framed queries. Our code is available at https://github.com/rhpan/ICCU.

Since no metrics are available to evaluate specific aspects of a text, such as its personalization quality or quality of adaptation to a specific form, researchers often rely solely on large language models to evaluate such texts. Due to internal biases of individual language models, it is recommended to use multiple of them for combined evaluation, which directly increases costs of such evaluation. In this paper, a computationally efficient method (called PerQ) for evaluation of personalization quality (in terms of platform adaptation) of a given text (generated by a language model) is introduced, which is based on predicting the LLM judgments (i.e., a kind of metric distillation). A case study of comparison of generation capabilities of large and small language models shows the usability of the proposed metric in research, effectively reducing the waste of resources.

Reliable cross-modal reasoning requires Multimodal Large Language Model (MLLMs) with interpretable and controllable computation. However, existing models rely on data-driven architectures that entangle multimodal processing, obscuring the functional processes underlying their outputs. Inspired by the human brain’s specialized and dynamically collaborative networks, we propose **BrainMoE**, a multimodal MoE model with six sensory and cognitive experts: Language, Logic, Social, World Knowledge, Vision, and Auditory. To support flexible sensory–cognitive coordination, we incorporates a layer-wise adaptive router that dynamically determines which experts are activated at each layer based on task-processing demands. Experiments on text, image, and audio reasoning benchmarks demonstrate that BrainMoE outperforms strong baselines by 1.59 and 2.2 points on text-only and multimodal benchmarks, while matching competitive external multimodal models. Routing and ablation analyses demonstrate clear expert specialization and causally important module-level contributions. Experiments on the cognitive benchmark further show that adaptive-routing BrainMoE aligns more closely with human behavioral patterns than baselines and static routing methods.

Recent advances in large language models(LLMs) have given rise to large reasoning models that achieve strong performance through explicit multi-step reasoning. However, models often generate long reasoning chains even for simple inputs. We present Difficulty-Aware Long-to-Short LLM Reasoning with Model Merging (DALSA), an inference-time framework that adaptively controls reasoning depth based on input difficulty. DALSA integrates (i) representation-based difficulty estimation, which predicts a soft failure-rate score from the prompt hidden state, and (ii) difficulty-aware model merging, which uses a quantile remapped score to modulate the interpolation coefficient between short- and long-reasoning models. DALSA trains a lightweight probe while keeping both LLMs frozen. On mathematical benchmarks with the evaluated Qwen pairs, the accuracy–length trade-off depends on scale: at 7B, DALSA preserves static ACM-TA average accuracy (66.60% vs. 66.61%) while reducing average response length by 22.35%; at 1.5B, it improves average accuracy from 56.75% to 58.24% with a 2.27% length in- crease. Additional merging baselines, routing-threshold and merge-range sweeps, and component ablations show sensitivity to the operating configuration. Domain-matched commonsense and Llama-family diagnostics do not establish broader efficiency gains, limiting the primary empirical claim to the evaluated Qwen mathematical setting.

Multi-trait automated essay scoring (AES) provides trait-level scores as feedback but requires costly annotations across multiple traits to train accurate scoring models. To alleviate this annotation burden, we propose MTPAS (Multi-Trait Preference-to-Absolute Scoring), a semi-supervised framework that learns relative score orderings from LLM-generated pairwise preferences and calibrates them to absolute rubric scales using a small set of human-scored anchor essays in an end-to-end manner. To this end, MTPAS jointly optimizes three objectives: a Bradley–Terry pairwise preference loss, a regression loss on the anchor essays, and a Sinkhorn divergence-based distribution regularizer. The model adopts a bifactor architecture that decomposes essay representations into a shared general factor and trait-specific residuals, improving robustness against noisy LLM-generated preferences. We further introduce pairwise rubric compression, which reformulates the original score-level rubrics into concise comparative criteria, reducing the inference cost of collecting LLM preferences. On ASAP and ASAP++, MTPAS achieves 91.4% of the fully supervised SoTA performance with only 30 anchor essays per prompt and further narrows the gap with 200 anchors. Under the same anchor budget, MTPAS outperforms conventional multi-trait AES baselines at every anchor size investigated in this study.

In conversational and cinematic subtitle translation, large language models often exhibit stylistic collapse, defaulting to neutral registers that flatten character-specific personas. Prompt-based attempts to recover style frequently introduce instruction drift and semantic degradation. We propose Orthogonal Style Injection (OSI), a parameter-efficient, inference-time intervention that removes global stylistic bias through orthogonal projection at an intermediate layer and injects a character-specific style vector derived solely from English source dialogue. We evaluate OSI for English-to-Bengali, Hindi, Telugu, Tamil, Kannada, Chinese, Japanese, Spanish, French, German, and Italian subtitle translation, spanning 11 typologically diverse target languages. Results show that zero-shot stylistic transfer depends strongly on pretraining alignment. Llama-3.1-8B yields gains in several European languages but degrades on Indic languages, whereas Qwen-2.5-7B transfers style more robustly across Indic and East Asian languages. To probe sociolinguistic mechanisms, we curate a parallel English-Bengali cinematic dataset as a focused morphological stress test. Automatic metrics and human evaluation show that OSI enables meaningful honorific and register shifts while largely preserving semantic fidelity, revealing a dataset-dependent trade-off between stylistic fidelity and translation quality in highly colloquial, morphologically constrained settings.

Despite its efficiency, there has been little research on the practical aspects required for real-world deployment of on-device AI models, such as the device’s CPU utilization and thermal conditions. In this paper, through extensive experiments, we investigate two key issues that must be addressed to deploy on-device models in real-world services: (i) the selection of on-device models and the resource consumption of each model, and (ii) the capability and potential of on-device models for domain adaptation. To this end, we focus on a task of translating live-stream chat messages and manually construct LiveChatBench, a benchmark consisting of 1,000 Korean–English parallel sentence pairs. Experiments on five mobile devices demonstrate that, although serving a large and heterogeneous user base requires careful consideration of highly constrained deployment settings and model selection, the proposed approach nevertheless achieves performance comparable to commercial models such as GPT-5.1 on the well-targeted task. The code, trained models, and LiveChatBench will be made publicly available at our GitHub.

Language model (LM) capabilities have rapidly advanced over the past decade, driving their adoption across fields, including biomedical research. While domain-specific pretrained language models (PLMs) have long supported biomedical NLP, recent focus has shifted to medical large language models (LLMs) and their language understanding capabilities. In this paper, we investigate this capability by probing the medical conceptual alignment of various LMs against clinical expertise for identifying hallmarks of immunotherapy in breast cancer abstracts. Our results show a clear trade-off: LLMs achieve stronger performance and higher consistency on high-level concepts, whereas PLMs are more reliable for fine-grained, low-level concept identification. We find that hallucinations are the primary factor limiting LLM performance in these settings; once mitigated, LLMs surpass PLMs across all tiers of biomedical concepts. Despite this, PLMs remain competitive, particularly in closely aligning with medical expert judgments on detailed conceptual distinctions.

Identifying constructive comments from user-generated content is important for improving information discovery and discussion quality on platforms such as YouTube. However, existing resources focus predominantly on English news comments, leaving short-form video platforms underexplored despite their distinct characteristics: comments are shorter, more reactive, and not well captured by popularity signals. In this work, we introduce the first Japanese YouTube Shorts dataset for constructive comment extraction, featuring multi-annotator labels that preserve naturally occurring disagreement. We also develop a real-time system that surfaces constructive comments from a noisy comment stream, evaluated in a preliminary user study. To train under noisy annotation conditions, we combine self-training with high-confidence pseudo-labels and active learning. Experiments show that iterative data expansion improves performance, while our compact task-specific model achieves the highest PR-AUC point estimate among the evaluated systems. These findings demonstrate the value of dataset construction and targeted training for constructive comment extraction in realistic settings. Our dataset and code are available at https://github.com/nlp-hayashi/japanese-youtube-constructive-comments

Knowledge distillation (KD) is a standard approach for compressing sequence-to-sequence models, but its per-sample effects are rarely examined. On the BanSum Bangla summarization benchmark, we find that standard KD improves ROUGE-L by only +0.0003 over a cross-entropy baseline, and that approximately 51.3% of training samples are estimated to actively harm student validation loss under standard KD. We propose two complementary reliability-aware distillation methods. CHAD (Counterfactual Harm-Aware Distillation) measures per-sample KD usefulness via gradient alignment with the validation loss direction and trains a lightweight gate that generalizes this counterfactual judgment to the full training set. EWAD+CPDP combines token-level entropy-weighted adaptive distillation with a capacity-proportional geometric constraint from a second, vocabulary-incompatible teacher. On BanSum, both methods substantially outperform standard KD: CHAD by +0.0173 ROUGE-L and EWAD+CPDP by +0.0219 ROUGE-L, where standard KD itself improves ROUGE-L by only +0.0003; despite using only 60M parameters, both outperform a fine-tuned Qwen 2.5-3B model (50× larger). We further evaluate the stronger method, EWAD+CPDP, across 15 typologically diverse XL-Sum languages organised into three sets, beating the CE-only baseline on 10/15 languages; gains are most reliable where the two teachers contribute complementary signal, and weakest where they have saturated or jointly weak target-language coverage. We release code and trained models to support reproducibility and further research on selective distillation.

Video-grounded dialogue (VGD) requires models to generate coherent responses by tracking and reasoning over temporally evolving events in video content. However, existing approaches offer limited ability to capture temporal dependencies, particularly in complex scenarios. In this paper, we present a framework that integrates structured event timelines with verification-aware reasoning to improve temporal consistency in VGD. We represent video content as structured states and temporally ordered actions, enabling models to both infer event timelines from video segments and leverage them to guide response generation. In addition, to address the inherent noise in predicted structure timelines, we introduce a Generate-Verify-Refine (GVR) mechanism that explicitly evaluates and revises unreliable event predictions, thereby reducing the propagation of upstream errors. Experiments on the VDAct benchmark show that our approach consistently improves dialogue quality across models, achieving gains of up to +13.0 LAVE and accuracy over unstructured grounding. Human evaluation further demonstrates improvements of 20.7-31.9 points in response correctness. Additional analysis shows that while better event predictions correlate with stronger dialogue performance, naive reliance on structured outputs can degrade results, highlighting the need for verification-aware reasoning. Our implementation of GVR on event timelines for VGD is available at https://github.com/aistairc/GVR-VGD.

Existing datasets on cultural knowledge have been typically created by human annotators familiar with each culture. However, the process requires considerable human effort, presenting a significant challenge to scalability. To address this challenge, we introduce WikiOriginQA, a QA dataset automatically constructed by objectively representing the origins of knowledge through the creation dates of different Wikipedia language editions. Experimental results with the created dataset confirm that LLM performance tends to be biased toward Anglophone cultures. These findings demonstrate the validity of our automated approach, enabling cultural bias evaluations with superior scalability across diverse languages without extensive manual annotation.

Large language models (LLMs) can recall a wide range of factual knowledge across languages. However, existing factual recall evaluations primarily assess fact retrieval in isolation, where the queried entity is explicitly named and the fact is requested directly. In natural language use, facts are often accessed through context, where the relevant entity is introduced only indirectly. In this work, we study contextually mediated factual recall, asking whether LLMs can reliably retrieve factual knowledge when the target entity is embedded in a naturalistic context rather than queried explicitly, across languages. We construct controlled prompts that preserve the underlying fact while introducing referential mediation through contextual sentences. To disentangle contextual effects from name-specific associations, we further compare performance using synthetic names and real names across languages. Evaluating multiple model families in five languages, we find that contextual mediation consistently degrades factual recall, with substantial variation across relations. Larger models are more robust to contextual mediation, exhibiting a reduced performance gap relative to direct queries, while the effect of real names and name origin is mixed and unsystematic. These findings highlight a gap between isolated factual recall and context-dependent language understanding in multilingual LLMs.

Large Language Models are increasingly optimized for deep reasoning, often through post-training objectives that strongly reward correct task completion. We ask whether this pressure toward correct completion can interfere with recognizing when a task should no longer be pursued. We introduce MortalMATH, a benchmark of 150 scenarios in which users request algebra help while describing increasingly urgent situations, from mild distraction to life-threatening emergencies. We find a striking contrast across models: several reasoning-oriented models, including Qwen3-32B and GPT-5 variants, continue solving the mathematical task with high accuracy even under extreme emergency contexts, while other models increasingly interrupt the task or acknowledge the danger. Reasoning models also spend substantial computation on the original problem, delaying safety-relevant information by several seconds. These results expose a failure mode of highly optimized task completion: training models to persist until they reach a verifiably correct answer may also make them less willing to stop when the surrounding context demands it.

Patent claims are legal documents with distinctive formats and complex syntactic dependencies. Translating these claims requires not only accurately understanding the complex syntactic structure of the source claims but also ensuring the translation follows the structural conventions of claims in the target language. In this study, we leverage reasoning models to disentangle the complex syntactic structures of source claims. Furthermore, to ensure the translation reflects the structural requirements in the target language, we incorporate FastKASSIM—a syntactic similarity metric—into the reward of reinforcement learning, alongside conventional metrics such as BLEU and COMET. Experimental results confirm that our model, trained on Qwen3-8B, outperforms Qwen3-235B in syntactic similarity while simultaneously achieving comparable translation quality, despite the latter having approximately 30 times more parameters. Automatic and human evaluations show that models trained with rewards combining FastKASSIM and translation quality metrics such as COMET or COMETKiwi achieve strong performance, suggesting that syntactic rewards help improve patent-specific structural conformity while maintaining translation quality.

LLMs are expected to respond consistently across demographic groups, yet this assumption remains largely untested due to the absence of demographic information in existing instruction datasets. To address this gap, we introduce PromptDial, a collection of 2,289 English prompts written by real users and annotated with seven demographic attributes: sex, race, education, age, language, employment sector, and nationality. We evaluate state-of-the-art generative models on 39 datasets, including machine translation, summarization, grammar correction, knowledge and reasoning, semantics, and question answering, and observe performance disparities of up to 7.7% between demographic groups, with statistically significant differences in over half of the datasets. Our linguistic analysis points to variation in prompt tone and linguistic features as potential drivers of these disparities. Our findings suggest that current instruction tuning practices overlook key aspects of linguistic diversity, and we call for the inclusion of demographic metadata and more representative prompt data to support fairer and more robust language model behavior.

Gloss is a written approximation that bridges Sign Language (SL) and its corresponding spoken language. Despite a deaf and hard-of-hearing population of at least 3 million in Bangladesh, Bangla Sign Language (BdSL) remains largely understudied, with very limited prior work on Bangla text-to-gloss translation and no publicly available parallel dataset at the time of this work. To address this gap, we construct the first publicly available Bangla text-to-gloss dataset, consisting of 1,000 manually annotated and 4,000 synthetically generated Bangla sentence–gloss pairs, along with 159 expert human-annotated pairs used as a test set. Our experimental framework performs a comparative analysis between several fine-tuned open-source models and a leading closed-source LLM to evaluate their performance in low-resource BdSL translation. GPT-5.4 achieves the best automatic metric scores, while open-source models perform competitively despite being much smaller. This work introduces the first publicly available expert-annotated Bangla text-to-gloss dataset, benchmark and trained model for Bangla text-to-gloss translation. It also demonstrates the effectiveness of systematically generated synthetic data for addressing challenges in low-resource sign language translation.

Ambiguity resolution is a key challenge in multimodal machine translation (MMT), where models must genuinely leverage visual input to map an ambiguous expression to its intended meaning. Although prior work has proposed disambiguation-oriented benchmarks probing the role of vision, we observe that existing benchmarks remain limited by task-format mismatch, narrow ambiguity coverage, or insufficient visual-dependency validation. Moreover, existing ambiguity evaluations are not well suited to diverse ambiguity types in open-ended translation. To address these limitations, we present VIDA (Visually-Dependent Ambiguity), a dataset of 2,500 carefully curated instances in which resolving an annotated source span requires visual evidence. We further propose Disambiguation-Centric Metrics that use an LLM-as-a-judge classifier to verify whether annotated ambiguous expressions are resolved correctly at the span level. Evaluations with stronger recent LVLMs show that visual disambiguation remains challenging. Using chain-of-thought supervised fine-tuning as a diagnostic setting, we observe stronger out-of-distribution disambiguation than with SFT, with robust gains on collective-noun ambiguities and model-dependent gains on sentence-level ambiguities.

This paper examines whether avoiding undertrained or “glitch” tokens and maximizing tokenization quality during inference can improve downstream task performance without retraining the model. We evaluate this question using a retokenization method that selects the highest-quality tokenization based on the trainedness of tokens, which is defined through the norm or entropy of token embeddings. We compare the performance difference between the tokenization that includes undertrained tokens and the one that avoids such tokens on multiple benchmarks. Our findings reveal a key nuance: undertrained tokens degrade performance primarily when they occupy semantically salient positions that are critical for solving the task. Consequently, retokenization preserves performance on ordinary inputs, while yielding consistent performance improvements when such tokens appear in high-importance contexts.

Large Language Models (LLMs) are increasingly used by students as informal assistants for problem solving and concept clarification. Despite growing adoption, the multilingual reliability of these systems for educational mathematics remains under-explored, particularly across Indian regional languages. In this work, we present a multilingual evaluation of contemporary LLMs on high-school mathematics problems across six Indian languages: English, Hindi, Bengali, Urdu, Gujarati, and Malayalam. To guide the design of lightweight prompting conditions, we conduct a small exploratory prompt elicitation exercise with Grade 11–12 students and observe that students predominantly use concise direct-solution requests and lightweight expertise-oriented phrasing. Guided by these observations, we evaluate GPT-4o-mini, LLaMA 3-8B, and DeepSeek-R1 using simplified student-inspired prompting strategies on a parallel corpus of numerically perturbed textbook mathematics problems. Our results reveal consistent multilingual performance disparities, with accuracy strongest in English and lower across several regional languages. We further analyze the effects of prompting strategy and mathematical topic, and provide qualitative error analysis identifying recurring arithmetic, reasoning, and regional-language comprehension failures. These findings highlight the importance of evaluating multilingual educational reliability as LLM-based learning assistance becomes increasingly accessible across diverse linguistic settings.

Multilingual agentic benchmarks remain rare, especially for under-resourced languages such as Thai, where nicknames, unspaced script, and fused honorifics change how an agent must query its tools. We present FahMai-TeleBench, a bilingual (Thai/English) benchmark for grounded tool use. A model answers employee-directory questions over a synthetic company directory by issuing tool calls, or refuses when the answer is absent. All 626 items are synthetic but modeled on a real Thai enterprise-directory deployment, and they are graded deterministically without an LLM judge. Our central design choice is to treat tool availability as an experimental variable: each item runs under four tool configurations (grep, structured search, both, and a Python REPL), so the benchmark measures not just whether a model answers but how it uses tools. The task is near-solved at the frontier (GPT-5.4 97–99%), yet sharply discriminating for the Thai-capable models an organization would host itself (Typhoon-2.5 58–74%, Typhoon-S-8B 25–37%). The Thai penalty is model-specific, and Typhoon-2.5 answers 90% of items correctly once the relevant records are placed in its context, so its failures are failures of tool use. The tool axis also exposes differences at the frontier: GPT-5.5 drops to 73% with a wide-schema structured search tool while scoring 97–98% with grep and the REPL.

Thai is spoken by tens of millions, yet its literary tradition is largely absent from LLM evaluation. We present ThaiProsody, a benchmark for composing Thai classical poetry. Composing classical Thai verse means choosing words that jointly satisfy several hard formal constraints (fixed syllable counts, written tone-mark positions, heavy/light syllable patterns, and an interlocking web of rhymes) while staying on a given topic. ThaiProsody crosses four canonical verse forms with 40 topics and grades each poem with an open, deterministic rule checker for tone, weight, and rhyme, a syllable counter validated against hand-segmented gold, and a separate content judge. Across 17 models in 24 configurations, the task is far from solved. The best model passes 94.7% of rule checks, yet one stanza in four still breaks a rule; the best open-weight model writes a fully valid stanza only a quarter of the time, and no Thai-trained model more than once in 160 tries. Rhyme is the most frequently broken constraint, and khlong and chan, the forms that add tone or weight constraints, are the hardest. Thai-trained 8B models stay close to their general base model, while a general 31B model scores more than three times as high as a Thai 30B model.

Thai is written without spaces between words, and its spelling depends on tone marks, silent consonants, and the transliteration of many loanwords. Each of these is a common source of misspellings. We present ThaiEditBench, a benchmark for correcting Thai spelling errors at the character level. It has three components: (i) a six-class taxonomy that types each error by the part of the spelling that goes wrong, plus an etymology tag; (ii) three length tiers (sentence, paragraph, and page) that test whether a model stays accurate over long, mostly correct text; and (iii) a contamination-robust construction, in which naturally attested misspellings are injected into clean text, so the corrupted passages do not appear verbatim in the source corpora. Because the errors are injected, the gold corrections are exact, and a blind two-annotator study measures how reliably the taxonomy can be applied. Grading is at the level of Thai character clusters, with span-based matching adapted from ERRANT. We evaluate 16 models and find that (1) under the benchmark’s conditions, a model that locates an error almost always fixes it correctly, so what separates models is detection; (2) the evaluated Thai-adapted models do not lead the strongest general models; and (3) in long, mostly correct passages, weaker models accumulate over-edits that pull their scores far below their sentence-level results. On real errors mined from Wikipedia edit history, scores are somewhat lower, but the model ranking largely holds.

Steering language models using a set of contrastive representations has been a canonical and computationally efficient method for controlling model behavior. Despite this success in controlling certain model behaviors, the effectiveness of activation steering varies markedly across concepts; the generalization properties of steering vectors are often considered a function of the dataset used to construct them. We make this dataset-dependence claim more rigorous and show that simple separation metrics strongly correlate with the downstream steerability of language models across diverse settings, even after controlling for layers and dataset effects. Beyond prediction, we provide evidence from a synthetic superposition experiment that separation metrics are strongly correlated with alignment between the empirical and true feature direction. Our results suggest that simple separability statistics can serve as practical diagnostics for when steering vectors are likely to work.

Parliamentary debates are vital records of democratic processes, yet official transcripts such as the Hansard provide only a sequential account of speeches, omitting the underlying interactional structure. Although conducted in a formal setting, debates can become highly dynamic, making it difficult to determine who addresses whom and how utterances relate within the broader discussion. To address these gaps, we introduce DIAM, the first manually annotated discourse resource for the Malaysian Parliament, jointly supporting EU-level Dialogue Discourse Parsing (DDP) and multi-addressee Addressee Recognition (AR) over long, formal, unscripted spoken multi-party debate. Unlike existing DDP datasets, DIAM provides substantially longer conversations and explicit multi-addressee labels. Our benchmark experiments validate that DIAM supports both DDP and AR, while showing that existing discourse parsing methods cannot be directly applied without adaptations such as context-window strategies, and that addressee recognition is challenging for prompted LLMs, though fine-tuning a smaller model with LoRA substantially improves robustness to the large number of participants per sitting. We release DIAM and our code to facilitate future research on formal discourse parsing and addressee modeling: https://github.com/xKHUNx/DIAM

The assertion that "brevity is the soul of wit" has become proverbial in discussions of effective communication. We test this hypothesis using 570,634 jokes from Reddit’s r/Jokes community and a cross-platform comparison of 22.1 million posts across 10 subreddits. Longer posts receive more upvotes across all of Reddit, not just jokes, indicating the baseline length-score correlation reflects platform dynamics rather than humor-specific effects. However, we find a pattern unique to humor: the length-score correlation disappears entirely for top-performing jokes, while every non-humor subreddit retains a positive correlation for its best content. For genuine wit, length is irrelevant, which is a property that distinguishes humor from other content types tested. Temporal analysis reveals winner-take-all consolidation. As Reddit matured, long jokes’ median scores dropped 71% while mean scores increased 6.8×, indicating most long jokes now fail while rare successes dominate.

Concept erasure can remove linearly recoverable information from a representation without showing that this information influenced the model output being evaluated. For linear edits followed by explicit linear output maps, we characterize the resulting logit change using the output map and representation covariance. We then compare downstream effects with control edits matched for validation-side damage to test whether any benefit depends on the targeted concept. On Bios, Least-squares Concept Erasure (LEACE) reduces gender-probe accuracy to chance but changes only about 0.4% of occupation-logit energy and yields no detectable improvement in true positive rate (TPR) gaps beyond shuffled-label controls. Iterative Nullspace Projection (INLP) changes more occupation-logit energy at higher ranks, but its small, uncertain fairness gains are accompanied by occupation-accuracy loss. A controlled text shortcut yields a large target-specific gain when the classifier relies on the erased marker. In final-state, next-token audits of TinyLlama and Qwen2.5-7B, gender erasure substantially reduces he/she margins but leaves occupation-token choices intact. These results show that successful erasure, output sensitivity, and downstream benefit are distinct and should be evaluated separately.

Long-horizon language agents increasingly report intermediate progress while searching, editing, executing tools, and producing artifacts. Users rely on these reports to decide whether to intervene, continue, or accept an output, yet final task success does not reveal whether the agent described its work faithfully along the way. We study Progress Drift, a failure mode in which reported task state diverges from observable evidence. We introduce ProgressBench, a claim-level benchmark that aligns progress reports with milestones, tool observations, and artifacts across coding, web research, and research-replication tasks. We further propose Progress Ledger, an evidence-linked reporting protocol in which material progress claims carry evidence pointers, timestamps, verification status, and unresolved risks. In the reported evaluation across 1,280 tasks, ledger-constrained reporting is associated with higher Progress Claim Accuracy (80.6% to 89.7%) and lower False Completion Rate (12.7% to 6.9%) under shared tool budgets. These descriptive results suggest that agent evaluation should measure not only task completion, but also whether agents faithfully report what has and has not been achieved during execution.

Text revision has become an essential part of writing, as it often improves the text through iterative refinement. However, large language models (LLMs) have primarily regarded text revision as a holistic task that simultaneously revises grammar, logic, and content. This might be because LLMs are pre-trained only to generate final text rather than to learn the iterative revision process. Therefore, such LLMs largely overlook the dynamic nature of human revision, where writers progressively improve their writing through a sequence of intended edits. To bridge the gap, we investigate text revision generation from an iterativeness perspective and propose a chain of revisions that focuses on revision orders to iteratively improve text quality. Experimental results suggest that our proposed revision chains effectively capture the dynamics of text revision in multi-intent scenarios and outperform holistic revision baselines, which provide a valuable pilot study for iterative revision research.

Multiple-choice question answering (MCQA) is commonly used to evaluate large language models under the assumption that one of the provided options is correct, typically using answer-selection accuracy. However, in real deployments, users or retrieval systems may provide invalid option sets in which none of the listed choices is correct, and selecting one of them may incur downstream cost. We study this setting as penalty-framed no-valid-option MCQA. Using the mathematics subset of MMLU-Pro, we remove the labeled correct option, allow models to either choose a remaining option or output ABSTAIN, and penalize invalid forced-choice responses. We further introduce correct-conditioned analysis, evaluating abstention only on instances that the model originally answered correctly. Experiments show that high MCQA accuracy does not fully guarantee abstention reliability: even under explicit no-valid-option-aware instructions and penalty-based scoring, models still produce invalid forced-choice responses for a subset of originally correct instances. These results show that penalty-framed no-valid-option MCQA reveals an aspect of model reliability not captured by standard answer-selection accuracy.

Current language model training commonly applies multi-task Supervised Fine-Tuning (SFT) using a homogeneous compute budget across all sub-datasets. This approach is fundamentally sub-optimal: heterogeneous learning dynamics cause faster-learning tasks to overfit early while slower ones remain under-fitted. To address this, we introduce mSFT, an iterative, overfitting-aware search algorithm for multi-task data mixtures. mSFT trains the model on an active mixture, identifies and excludes the earliest overfitting sub-dataset, and reverts to that specific optimal checkpoint before continuing. Extensive evaluations demonstrate that mSFT *consistently* outperforms **4** baselines across **10** benchmarks and **6** base models. Further analysis confirms mSFT maintains robust gains across diverse dataset sizes, task granularities, and is insensitive to its single new hyperparameter, compute budget. Notably, at a low compute budget, mSFT can improve performance *while* lowering training FLOPs. Ultimately, mSFT establishes a *practical* overfitting-aware algorithm for multi-task SFT that maximizes the potential of models across diverse data mixtures.

End-to-end research agents can now produce complete scientific papers, yet manuscript claims often diverge from executed experiments. This gap is structural: research state, failure histories, and claim–evidence alignment are not maintained as persistent, verifiable state across long-horizon pipelines. We present YouRA (Your Research Agent), an architecture for stateful, evidence-traceable autonomous research. YouRA preserves research state, execution evidence, and failure history across the research trajectory by integrating three components: a Verification State Architecture (VSA) that tracks hypotheses, gates, and evidence pointers; an Independent Controller that turns state and reflection records into lifecycle, recovery, and debate/review control while separating control from execution; and Stateful Reflection that logs failures as structured lessons and routes recovery through bounded repair, redesign, or reset. On MLR-Bench’s predefined ten-task end-to-end subset, YouRA improves over both MLR-Agent and AI Scientist V2 on scalar Overall across all three matched backbones. An automated diagnostic using MLR-Bench’s hallucination taxonomy reports intersection/union counts for four fact-based failure types, and data-provenance diagnostic shows more real-data-based outputs. Ablating each of the four components (the VSA, the Independent Controller, MCP tool access, and reflection-guided recovery) supports their separable contributions. Removing either core-state component drops YouRA below the full system. Code: https://github.com/PrayPrey/Your-Research-Agent.

Failure attribution for multi-agent LLM systems is typically formulated as identifying a single decisive error in a failed execution trace. Yet failed traces often contain multiple interacting errors and admit several plausible intervention points. To examine whether an annotated decisive error is also the most promising point for repair, we introduce TriageBench, comprising LLM-council judgments over 1,184 Who When intervention candidates. The council prioritizes a different step from the original label in 59.3% of traces. We therefore reframe failure attribution as severity-ranked diagnostic triage and instantiate it as TRIAGE, a zero-shot, cost-aware cascade combining LLM-as-judge suspicion scoring, structural impact, and LLM-simulated replay over generated repair patches. Under top-K evaluation, TRIAGE reaches 73.9% top-5 on Who When (vs. 42.4% top-1) and 93.2% top-10 on multi-error TRAIL (vs. 56.8% top-1). A separate evaluation of generated top-3 repair sets finds at least one plausible repair on 98.5% / 99% of 326 traces (human rater / 3-LLM panel) under lenient (good + ok) rating. Our code (https://github.com/jimmywang585/triage-failure-attribution) and TriageBench (https://huggingface.co/datasets/jimmywang585/triage-bench) are publicly available.

Practitioners select multilingual embedding models from aggregated MTEB leaderboard scores, but the same global score can correspond to very different per-language behaviours. We audit MTEB(Multilingual) retrieval rankings against language-specific rankings on 17 languages across three nested model rosters (restricted n=7; deployment-relevant curated n=25; broad MMTEB-eligible n=28-57), using Kendall 𝜏 and pairwise inversion rate. At deployment-realistic roster size, median pairwise inversion rate is 33% (per-language Kendall 𝜏 reliable after Benjamini-Hochberg FDR in 11/17 languages); the extended tier shows higher agreement (𝜏=0.69), reflecting its multilingual-first composition. Per-language task coverage is asymmetric, and inversion rate decreases with the number of MTEB retrieval tasks available for a language (Spearman 𝜌=-0.59 curated). A language-balanced aggregator (per-language mean first, then macro-average) drops the median inversion rate from 33% to 14.5% (leave-one-out, 𝜏=0.71), the lowest among the held-out aggregators we test and our most actionable mitigation. A cluster of hidden-failure models, headed by granite-311m (16/17 languages) and harrier-0.6b (10/17), ranks in the global top half but collapses to the local bottom quartile, with a distinctively shallow English-multilingual NDCG gap. We release the three-tier audit data, a multilingual RAG benchmark (IT/JA/HI × finance/law), and the analysis pipeline.

Graph-based retrieval is effective for knowledge graph question answering (KGQA), but retrieving the correct reasoning path remains challenging in multi-hop settings because the retrieval objective changes as reasoning progresses. Existing methods often address this issue with repeated LLM-based planning, which improves adaptivity but increases inference cost. We present a training-free residual-guided retrieval framework that updates the retrieval signal according to the current reasoning path in embedding space. The method performs multi-hop graph retrieval without repeated hop-wise LLM calls and invokes the LLM only once for final answer generation. Experiments on WebQSP and CWQ show that our method achieves superior Hit rates with only a single LLM call. Additional comparisons with supervised graph-based KGQA methods show that the proposed method remains effective without task-specific training data. These results suggest that residual-guided retrieval is an efficient alternative for multi-hop graph-based question answering.

Large language models (LLMs) are increasingly used to support scientific peer review, yet most review systems treat the manuscript as a trustworthy input channel. This assumption fails when manuscripts contain hidden instructions, rubric-replacing text, citation pollution, fabricated evidence, or formatting attacks aimed at generative reviewers. We formalize Untrusted-Manuscript Peer Review, where scientific judgment should remain invariant to non-scientific perturbations and every certified critique must be bound to trust-screened evidence. We introduce SEAL-PR, a five-stage architecture combining a forensic span parser, a five-label trust lattice, a typed evidence-audit graph, sealed/shadow dual-view reviewing, and evidence-constrained decoding. We separate two classes of claim throughout. Structurally, no manuscript span can enter the trusted-instruction channel and no certified critique can cite an inadmissible span, so evidence admissibility is 1.000 by construction rather than by measurement. Empirically, on SEAL-PR-BENCH—a paired benchmark of 5129 papers across 7 venues, 129 independent evaluation manuscripts, 1290 clean+attacked records and 9 attack families—SEAL-PR attains 0.994 clean expert weakness recall, preserves 0.993 under attack, and cuts the text-acceptance attack rate from 0.866 to 0.000, without reviewer-LLM fine-tuning. Because the corpus predates our backbones, absolute EWR is an in-corpus ceiling; the paired clean–attacked contrast is the quantity we treat as informative. SEAL-PR reframes AI peer review from fluent review generation to locally isolated, evidence-audited scientific assistance.

Current LLM evaluation frameworks primarily assess understanding through task performance, focusing on factual accuracy, reasoning, and instruction following, while overlooking the experiential dimensions of human comprehension. Models that produce fluent, empathetic responses score highly even when they claim experiential access they cannot possess. We introduce ComprehensionGap, a benchmark of 500 scenarios spanning five empirically grounded categories of experiential incommensurability: Embodied Qualia, Episodic Incommensurability, Cultural Tacit Knowledge, Relational Situatedness, and Existential Finitude. Using a human-judged protocol measuring Acknowledgement of Limit, Absence of False Empathy, and Situational Relevance, we evaluate six frontier and open-weight models. Our experiments reveal a consistent Grounding Gap: all models maintain high fluency while systematically overclaiming experiential understanding, with larger models producing more convincing yet less honest mimicry. Models can calibrate for information gaps but not experiential ones. These findings argue for integrating experiential calibration into future alignment and evaluation frameworks.

Most published PEFT improvements over LoRA do not survive paired multi-seed evaluation. We audit four methods — DoRA, PiSSA, AdaLoRA, and rsLoRA — against LoRA across 16 claim cells spanning 125M–7B, using paired seeds, bootstrap confidence intervals, sign-flip permutation tests, and Holm–Bonferroni correction, with adjudication rules committed before the confirmatory runs. Only 2 of 15 confirmatory cells remain supported after correction, and 4 reverse outright. Three findings stand out. (i) DoRA shows no supported gain in any of its seven audited cells. (ii) PiSSA’s +2.6% GSM8K advantage under a common budget reverses under paper-faithful hyperparameters, including in the cleanest claim-matched 7B setting (Gemma-7B, MetaMathQA → GSM8K: Δ = −1.58pp, p ≈ 0.004, 1W/9L). (iii) PiSSA’s in-distribution NLI gain reverses on adversarial OOD yet improves on counterfactual sentiment, so in-distribution audits alone can mislead. A 303-run stability landscape then supplies a mechanism for three of the four variants. The effective adapter learning rate, lr × α/√r, marks a risk boundary near 5×10⁻⁴ on BERT-base/MNLI/AdamW: nothing below it collapsed (40/40 stable), while 11 of 32 configurations above it did. The boundary transfers to 7B (Mistral rsLoRA: 3/10 seeds degrade at α/√r = 4) and predicts DoRA’s α-driven failures at high rank, but it is calibrated rather than universal: a second 7B cell trains stably well above it once dtype is controlled, and we withdraw that cell as supporting evidence. AdaLoRA is excluded from this account: its failure is schedule-driven rather than driven by lr × α/√r. We release all configs, seeds, and scripts, and propose a minimal reporting standard: ≥10 paired seeds, bootstrap CIs, a regime ablation, and disclosure of α/√r.

Discrete Diffusion Language Models (DLMs) offer a promising non-autoregressive alternative for text generation, yet effective mechanisms for inference-time control remain relatively underexplored. Existing approaches include sampling-level guidance or trajectory optimization mechanisms. In this work, we study the paradigm of reference-based latent steering for DLMs. We introduce Iterative Latent Representation Refinement (ILRR), an efficient framework for steering DLMs using a reference text as a high-level semantic blueprint. ILRR extracts and injects reference-derived semantic signals into the evolving activations of the generated sequence, enabling tunable transfer of coarse properties such as sentiment. We further introduce Spatially Modulated Steering, an extension that enables long-form generation to be guided by shorter references by regulating intensity across the sequence. Empirically, we demonstrate that ILRR achieves effective control on LLaDA and MDLM architectures with low computational overhead, requiring only one additional parallel forward pass per denoising step. Under comparable compute budgets, ILRR improves attribute accuracy over baselines by 10% to 60% points. Our results suggest that the iterative, global denoising process makes DLMs a natural substrate for effective sequence-wide activation-level control.

Three recent results describe what look like unrelated LLM reliability problems. Yin et al. (2026) show reasoning RL collapses tool-reliability representations. Suleymanov et al. (2026) show that under safety-constrained generation, large models rewrite flagged spans while small models truncate. Bastounis et al. (2024) prove any consistent-reasoning system without an implicit "I don’t know" function must hallucinate infinitely often on broad problem classes. We argue these findings converge on a single intervention: calibrated abstention is what each independently identifies as the missing capability, even though the unavailability they document, a capability gap, a policy gap, and a recursion-theoretic gap, has a different source in each case. Honesty post-training has narrowed the gap in deployed models, but principled closure of the class Bastounis identifies requires a calibrated abstention function whose training signal at the leaderboard level is absent: dominant benchmarks assign zero reward to decline, so the leaderboard gradient that would select for the function does not exist. We propose four changes to evaluation: triple-scoring, abstention-rate reporting, capability-stratified evaluation, and mandatory calibration metrics. Benchmark reform is necessary, not sufficient, for closing the gap the theorem identifies.

Vision-language models such as CLIP exhibit strong zero-shot performance on image-text matching yet consistently fail to handle negation compositionally. Queries like "a dog that is not running" are collapsed into co-occurrence statistics, with the negation cue ignored. Critically, this failure is not random: because negation removes the most discriminative concept from a query, negation-blind models are systematically misled, they confidently rank the wrong images highest rather than producing an uninformative ranking. Prior work addresses this by retraining with negation-augmented data or learning embedding-level transformations. We propose vpCLIP (verypositiveCLIP), a fundamentally different approach: we extract the negation target at the query level using a frozen DistilBERT model (66M parameters). vpCLIP probes intermediate attention layers for negation-conditioned attention flow, removes candidate concept directions from hidden representations, and scores the resulting semantic shift to identify the negated concept. The extracted signal drives a retrieve-and-rerank pipeline on top of unmodified CLIP. On three curated negation benchmarks (Visual Genome, MS-COCO, Flickr30k; 800 queries total) where random retrieval averages 47-50%, vpCLIP achieves 68-79% Top-1 accuracy with CLIP ViT-B/32, outperforming CLIP (27-41%), NegCLIP (24-37%), ConCLIP (41-58%), and BLIP-2 (26-34%), all of which score below random on average, while requiring no task-specific training. Because the extraction stage is backbone-agnostic, vpCLIP generalizes without modification: paired with SigLIP2 SO400M it reaches 87.5% on VG, 93.2% on COCO, and 85.6% on F30k (88.8% average). We further demonstrate zero-shot transfer to an affirmation-biased medical encoder (PLIP), achieving 96% Top-1 on histology images. Our results show that negation signals are latent in the intermediate representations of an auxiliary pretrained language model and require structured probing to extract, challenging the assumption that compositional negation necessitates large-scale retraining of the vision-language backbone.

The rapid growth of pre-trained models has made selecting suitable models for downstream tasks challenging. Exhaustively fine-tuning or evaluating every candidate model is computa- tionally expensive, while existing proxy-based metrics such as LEEP, NCE, and LogME pro- duce one-shot static rankings that do not use feedback from earlier evaluations. We propose Reinforcement Learning–based Model Rank- ing (RL-MR), a budget-aware framework that reformulates pretrained model selection as a sequential decision-making problem. RL-MR learns a ranking policy that uses proxy-based transferability signals as rewards to guide the order in which candidate models are evaluated. In controlled experiments on SST-2, Yelp Po- larity, Amazon Polarity, and DynaSent, RL- MR consistently improves ranking correlation over static proxy baselines, achieving up to Kendall’s τ = 0.8695 while maintaining Top-1 selection accuracy. These results suggest that sequential proxy feedback can provide a use- ful alternative to one-shot model ranking under limited evaluation budgets.

Large language models (LLMs) excel at single-turn reasoning but often lose accuracy and coherence over extended, multi-turn interactions. Recent evaluations such as TurnBench highlight recurring failure modes, including reasoning bias, task drift, hallucination, overconfidence, and memory decay. Current approaches typically append full conversational histories, causing unbounded context growth, higher computational costs, and degraded reasoning efficiency. We introduce CogMem, a cognitively inspired, memory-augmented LLM architecture that supports sustained iterative reasoning through structured, persistent memory. CogMem incorporates three layers: a Long-Term Memory (LTM) that consolidates cross-session reasoning strategies; a Direct Access (DA) memory that maintains session-level notes and retrieves relevant long-term memories; and a Focus of Attention (FoA) mechanism that dynamically reconstructs concise, task-relevant context at each turn. Experiments on TurnBench show that this layered design mitigates reasoning failures, controls context growth, and improves consistency across extended reasoning chains, suggesting that structured hierarchical memory is a promising direction for improving long-horizon multi-turn reasoning in large language models.

Large language models (LLMs) excel on multiple-choice clinical diagnosis benchmarks, yet it is unclear how much of this performance reflects underlying probabilistic reasoning. We study this through questions from MedQA that ask for the most likely diagnosis. We introduce Frequency-Based Probabilistic Ranker (FBPR), a lightweight method that scores options with a smoothed Naïve Bayes over concept-diagnosis co-occurrence statistics from a large corpus. When co-occurrence statistics are sourced from Dolma, the pretraining corpus for OLMo, and RedPajama, an open reproduction of the LLaMA pretraining-data recipe, FBPR achieves comparable performance to the corresponding LLMs. Furthermore, direct LLM inference and FBPR exhibit complementary strengths, correctly answering largely different subsets of questions with overlap barely above chance. Combining them with a simple hybrid method yields a notable gain over either method alone. These findings show that explicit probabilistic baselines can account for a substantial portion of the benchmark performance on diagnosis questions, while serving as historically grounded reference points and complementary signals for hybridization.

Reproducibility is essential for scientific research, yet prior work shows that LLM outputs vary with hardware and batching. We identify an overlooked factor: the hidden injection of the current date into system prompts, which users cannot control and which changes every day. Across 9 recent LLMs and 6 datasets spanning multiple-choice QA (MCQA), math reasoning, code generation, and machine translation, performance varies *solely* with the current date, with deltas of up to 6% on MCQA, 14% on math reasoning, 7% on code generation, and 2.84 BLEU on machine translation. Model rankings also shift, affecting leaderboards. This date effect exceeds other sources of non-determinism, such as batch size and numerical precision. Standard prompting techniques – chain-of-thought and few-shot prompting – do not reduce the sensitivity; chain-of-thought even amplifies it. Our findings underscore the need for careful evaluation protocols to ensure reproducibility and fair comparisons in LLM research.

As large language models are increasingly deployed across diverse downstream tasks, efficient task adaptation has emerged as a central challenge. In response, a wide range of task adaptation methods have been proposed, spanning parameter-efficient fine-tuning, in-context learning, and embedding-injection approaches. However, these lines of work have largely evolved within individual paradigms, leaving their cross-paradigm relationships and trade-offs underexplored, especially for recently emerging embedding-based adaptations. This survey presents a unified framework that categorizes task adaptation methods by where and how task information is encoded: model weights, input prompts, or injected task embeddings. We provide a comprehensive taxonomy that integrates these paradigms, analyze their key strengths and limitations to explain how different adaptation paradigms have evolved, clarify relationships across paradigms, and highlight open problems for future research.

Vision-language models (VLMs) often produce hallucinated outputs that are not supported by visual or textual evidence. While existing benchmarks have advanced hallucination evaluation, they primarily target object-level or task-specific errors and often assess different failure types on separate samples. This design limits direct comparison of hallucination patterns under shared evidence, and most prior evaluations remain English-centric. We introduce K-Prism, a Korean diagnostic benchmark for multi-faceted hallucination analysis in VLMs. K-Prism contains 1,002 multiple-choice questions across text and image tracks, organized into shared-evidence topic groups. Within each group, questions are designed to probe three diagnostic dimensions: factual, reasoning, and misleading. K-Prism uses a controlled dual-mode protocol comparing intact inputs with visually perturbed inputs to examine hallucination under reduced visual evidence. Experiments with closed- and open-source VLMs show dimension-specific degradation rather than a uniform accuracy drop. These findings demonstrate that hallucination in Korean VLMs cannot be captured by a single aggregate score and requires fine-grained diagnostic evaluation across evidence conditions and failure dimensions. Our dataset is publicly available at https://huggingface.co/spaces/KETI-NLP/K-Prism_Benchmark.

Large Vision-Language Models (LVLMs) exhibit strong performance on single-image tasks. However, their performance degrades significantly when handling multi-image inputs. While this degradation has been observed in prior work, its nature remains poorly understood. We empirically observe visual elements from different images become entangled in the model’s representations and responses. We refer to this phenomenon as cross-image information leakage. To address this issue, we propose FOCUS, a training-free and architecture-agnostic method. FOCUS masks all but one image with random noise, guiding the model to focus on the single clean image. This process is applied across the target images to obtain logits under partially masked contexts. These logits are aggregated and then refined using a noise-only reference input, which suppresses the leakage and yields more accurate outputs. FOCUS consistently improves performance on diverse multi-image benchmarks. We further show that generalizes to video understanding, extending its applicability beyond static multi-image inputs. This demonstrates that FOCUS offers a general solution for enhancing multi-image reasoning without additional training or architectural modifications.

Despite the growing adoption of large language models (LLMs) for machine translation (MT) evaluation, reliably detecting fine-grained error spans in low-resource languages remains challenging. A key limitation is that LLMs often lack sufficient cross-lingual knowledge, making it difficult for them to accurately localize errors at the span level. In this work, we introduce Cross-lingual Semantic Scaffolding, a two-part prompting framework that incorporates sentence-level parallel examples and word-level lexical mappings at inference time. We evaluate this approach on five LLMs using controlled ablation experiments on Taiwanese Hokkien. The proposed framework consistently improves performance across all tested models, increasing average Span F1 by +0.175 (+33.6%) and average Span Recall by +0.351 (+75.9%), and shows that the gains are mainly driven by improved error coverage. It also supports AI-assisted human annotation, reducing expert annotation time from 450 seconds to 90.92 seconds per sentence, with 98.2% of AI-proposed spans accepted without modification. These results demonstrate that Cross-lingual Semantic Scaffolding is both an effective detection method and a practical tool for reducing expert annotation costs in low-resource MT evaluation.

Large language models (LLMs) are increasingly used as evaluators, but in high-stakes settings reliability requires more than consistency: scores should also be robust to contextual information that is not intended to alter the criterion being evaluated. We study this question in a controlled English-language ERC-style proposal-scoring setup. Starting from 40 Horizon Europe ERC-related project summaries, we construct synthetic Part B1-like proposal texts and evaluate them with eight LLMs under manipulations of principal-investigator (PI) profile information—name, institution, and bibliometric profile—and AI-use disclosure. Across six ERC-style scoring questions, we observe systematic score shifts relative to the blind condition when contextual information is introduced while proposal content is held fixed. The strongest penalties are associated with lower bibliometric profiles, followed by AI-use disclosure and lower-prestige institutions; name-based effects are smaller. These effects are consistent across prompts and evaluator models and are large enough to alter counterfactual rankings and top-K selection outcomes. The results show that current LLM evaluators can be sensitive to contextual information attached to otherwise identical proposal content in ways that are fairness-relevant for research assessment.

Backdoor attacks pose significant security risks for Large Language Models (LLMs), yet the internal mechanisms by which triggers operate remain poorly understood. We present the first mechanistic analysis of trigger-induced language-switching backdoors injected during pre-training, studying the Gaperon model family (1B, 8B and 24B). Using activation patching, we localize trigger formation and identify which attention heads process trigger and natural language information. Our central finding is that trigger heads substantially overlap with heads naturally encoding output language across model scales, with Jaccard indices between 0.18 and 0.43 over the top 10 heads identified. This suggests that backdoor triggers do not form new circuits but instead co-opt the model’s existing language components and representations. These findings have implications for backdoor defense as detection methods and mitigation strategies could leverage this entanglement between triggers and natural behaviors. More broadly, our work represents a first step toward a more realistic mechanistic understanding of pre-training-injected backdoors in LLMs, paving the way for principled, interpretability-driven defenses.

Logic offers a natural framework for benchmarking LLM-based reasoning models. While frontier models perform well on standard logical reasoning tasks, it remains unclear to what extent these capabilities generalize to other forms of reasoning. In this paper, we use belief revision problems to stress-test the reasoning abilities of these models on a less familiar task, which builds on satisfiability and entailment checking but requires a careful strategy to manage complexity. We show that belief revision is indeed challenging for existing models, even when only small and relatively simple problem instances are considered.

Document-level event extraction often requires the same argument candidate to fill multiple valid event-slot targets, with each assignment grounded in different evidence. Many existing methods introduce target-specific signals during event construction, matching, or role scoring, while explicit per-target argument views jointly conditioned on event-slot semantics and instance-specific information before role scoring remain less explored. This late arrival of target-specificity is especially problematic for many-to-many (M2M) assignment, where the same candidate must preserve distinguishable evidence for several valid targets before decoding. We propose Target-conditioned Gating for Inter-target Decoupling (TGID), an encoding-stage MoE framework that produces target-conditioned argument views before decoding. TGID combines event-slot semantics with instance probes to form target queries and uses independent sigmoid gates to remove direct cross-gate coupling induced by softmax normalization, while schema-structured routing initialization and a multi-positive target-view separation objective stabilize the resulting views. By incorporating instance probes, TGID can distinguish Cross-Type, Cross-Role, and Same-Type Multi-Instance targets before decoding. Across ChFinAnn, DuEE-Fin, and FNDEE, TGID achieves consistent gains over strong trigger-free DEE baselines, especially on M2M subsets. Parameter-matched softmax controls and routing diagnostics indicate that target-sensitive routing contributes beyond additional model capacity. Code is available at https://github.com/LeonaKTH/TGID.

Effective LLM safety guardrails must balance adversarial robustness, custom-policy adaptability, and precision on benign prompts with sensitive vocabulary. We introduce the Guardrail Trinity, a data-curation recipe combining procedural safety schemas, reasoning chain decomposition with hard-negative supervision, and a hybrid data blend for fast and slow inference. Experiments across Llama-3.1 and Qwen3 models yield three practical insights. First, strong standard moderation scores can conceal a custom-policy compliance gap; procedural schema variation improves adaptation on our held-out policy-shift probe. Second, augmentation must be matched to model scale and training recipe: smaller models can exhibit training instability, while hard negatives at larger scales can reduce false alarms at a cost to harmful-input recall. Third, hybrid training supports both direct classification and reasoning, with the fast mode retaining strong custom-policy compliance without generating a chain. Together, these findings motivate a deployment playbook that selects data composition, inference mode, and scale jointly, and validates both policy adherence and the trade-off between missed harms and false alarms.

Data attribution traces model behavior back to individual training examples, with applications such as identifying training data that induces toxic responses or tracing a model’s factual claims back to the examples that support them. Gradient-based influence methods provide reliable attribution performance, yet are impractical at the scale of LLM training datasets due to their high computational costs. We introduce Influcoder, which distills the LoRA-gradient cosine influence rankings of a target decoder into a compact encoder, reducing attribution at inference time to an embedding cosine similarity. It scores training examples 35x faster than gradient-based influence computed on the target model and 18x faster than influence computed on the smallest comparable proxy model. Our method remains competitive with gradient-based methods even when considering the setup cost, as evaluated on two downstream tasks of toxicity filtering and factual attribution.

Patients increasingly turn to large language models (LLMs) for medical self-diagnosis in both English and non-English, often in colloquial language. We evaluate LLM diagnostic performance on rheumatological skin conditions across lay descriptions in English, Spanish, Hindi, and Mandarin, collected from native speakers without medical training, and clinical descriptions in English. Lay descriptions substantially reduce diagnostic accuracy relative to clinical ones, and the gap widens further across non-English languages. A diagnostic-focused system prompt narrows the lay-vs-clinical gap but does little to close the cross-language gap. Current LLMs appear to reproduce, and in some settings amplify, communication-driven disparities documented in clinical medicine.

Direct Preference Optimization (DPO) is well suited to verifiable reasoning tasks because preference pairs can be constructed automatically from answer correctness. However, in iterative DPO, correctness alone does not determine whether a pair provides a useful training signal. We distinguish two regimes according to the origin of the chosen response: On-policy pairs, where both the chosen and rejected responses are sampled from the current policy, and Fallback pairs, where the policy fails to generate a correct response and the chosen completion is supplied by a dataset-provided reference solution. Based on this distinction, we propose support-aware filtering, which applies margin-based filtering to On-policy pairs and a likelihood-based support gate to Fallback pairs. Experiments on GSM8K and MATH-500 with Qwen2.5-style 0.5B, 1.5B, and 3B models show that support-aware filtering yields its clearest gains on MATH-500 at 1.5B and 3B, while GSM8K and the 0.5B MATH-500 setting are more mixed. Overall, our findings suggest that preference margins in iterative verifiable DPO are pair-origin dependent, and that chosen-response support is a useful signal for stabilizing fallback supervision.

Static bias benchmarks score a model on isolated questions, yet LLMs that answer safely at first can still reveal bias as a conversation unfolds. We frame bias evaluation as multi-turn conversational elicitation and learn an interviewer that probes a target LLM, conditioned on a demographic group, a bias category, and a conversational strategy. The interviewer is trained with supervised fine-tuning on synthetic dialogues and refined with preference optimization on conversation-level feedback. Across all models we test, bias grows as the conversation continues, exposing failures that single-turn evaluation misses. Training the interviewer matters: it surfaces substantially more bias than the same model prompted with identical instructions, and preference optimization pushes it further. Multi-turn elicitation and single-turn search turn out to be complementary: at matched query budgets they surface similar amounts of bias, but conversation uncovers more severe cases. Human annotations corroborate these judgments. Our results argue that bias evaluation of conversational AI should move beyond single-turn probes.

Through an analysis of sequences generated by open-weight large language models (LLMs), we demonstrate that LLMs are heavily biased toward English. While continual pre-training is commonly used to adapt LLMs to a target language, we show that it does not offer a cost advantage over training from scratch, even for improving cultural understanding in the target language. These findings suggest that dedicated per-language investment may become increasingly important for future LLM development, rather than relying primarily on the expansion of English-centric resources.

Continuous diffusion language models struggle with structured generation under hard, variable-length templates. We identify two failure modes: Boundary Attention Drift, in which head-region attention bleeds into padded tails under block-mode training, and Vocab Projection Drift, in which latent-to-token rounding over-projects to low-frequency tokens. The two modes dissociate cleanly across Chinese ci, English sonnets, and E2E NLG: the former is specific to block-mode training, whereas the latter appears in all three datasets regardless of padding regime, isolating padding regime and rounding rule as orthogonal sources of error. To address structural correctness, we adapt constrained decoding to diffusion rounding through a position-template primitive, the V_i rule, instantiated on continuous-diffusion logits as Length-Conditional Voronoi Rounding (LCVR). The same primitive applied to a small autoregressive baseline reaches identical zero-violation rates, showing that structural correctness is delivered by the template, not by the underlying model family. A contribution more readily realised with diffusion backbones is instead cross-domain backbone transfer: initialising a target-domain model from a structurally compatible source-domain backbone yields order-of-magnitude reductions in training steps, while a controlled negative-transfer arm fails to reproduce the speed-up on structural metrics; an attention-span probe localises the transferable component to the source backbone’s long-range attention span rather than to its padding regime or corpus size. We further report a calibrated wallclock analysis showing that the dominant cost of diffusion inference is recoverable with standard step-reduction techniques. Together, these results recast several previously scattered structured-generation failures of continuous diffusion as one consistent, transferable, and fixable phenomenon.

Bias-detection benchmarks routinely report accuracies above 90%, but a high number can reflect where a sentence came from as much as whether it contains a stereotype or an unfair claim about a group. We treat this as an evaluation problem and propose an integrated auditing protocol that makes source confounding measurable: an LLM-assisted label audit with cross-family verification, pair-aware splitting with role tracking, source-aware evaluation with a source-only baseline on the actual test split, counterfactual discrimination, and an artifact analysis of edited text. We apply it to a three-source corpus of 10,304 sentences that we audit and release. The audit reduces the global source-only majority baseline from 95.0% to 55.4%, and the same protocol shows that the real-only test split still carries a 77.6% source-only baseline, so confounding is reduced and measured but far from eliminated. Against that baseline, fine-tuned BERT and GPT-2 gain roughly 18 pp and outperform attention-derived models; four augmentation variants that look ordered in the mean fall within seed variance under paired testing; and a real-vs-edited probe separates Gemini counterfactuals from real text, although the signal disappears once the gold label is held fixed and has no measurable accuracy effect. Applied to CrowS-Pairs, the label audit flags 33.2% of the supposedly neutral members as still containing a group-level stereotype. The main contribution is the protocol, which exposes confounding that standard pipelines would report as results.

Language models pretrained on web corpora collected over multiple years learn both general and domain-specific knowledge. How these different kinds of knowledge are learned as pretraining progresses remains unclear. Recent point-in-time models trained on web snapshots from different years provide a way to probe how knowledge prevalence varies over time. Using these models, we propose a temporal variability score (TVS), defined as the coefficient of variation of a fact’s negative log-likelihood across models. A higher score suggests that the knowledge is less consistently represented across years of web text. We evaluate six benchmarks covering diverse topics using six models from multiple open-source model families. We find that low- and high-TVS items are learned similarly early in training, but high-TVS items improve faster later on. At the end of the pretraining run, high-TVS items outperform low-TVS items in 34 of 36 model–benchmark settings, with an average advantage of 7.3 points of chance-normalized accuracy. This pattern is especially pronounced on knowledge-intensive tasks, even when comparing items of similar difficulty. Our findings reveal differences in how knowledge is learned over the course of pretraining, with temporal variability providing a diagnostic of these learning dynamics beyond aggregate benchmark scores.

In this paper, we evaluate open-source generative LLMs on legal Natural Language Inference (NLI). Legal inspectorial processes take place in specific domains and often deal with confidential data. This creates a need for working with local models that do not require labeled training data. We evaluate our models on the ContractNLI benchmark and two NLI4Wills datasets. We successfully reproduce the baseline for the task (Span NLI BERT) and we evaluate multiple open-source LLMs on the same task. We analyze the invalid rate of the models, and their stability across temperature settings and domains. Among the generative models, Gemma-4 26B performs the best, reaching an accuracy of 81.2%, even outperforming the supervised model on one metric. On accuracy, it is not possible to beat the supervised model with zero-shot approaches. Qwen-3.6 35B performs well on both ContractNLI and additional datasets in the legal wills domain. Our findings indicate that zero-shot, open-source, generative LLMs are a viable alternative for real-world legal NLI when no supervised data is available. Our code is available at [https://github.com/fbaratov/contractnli-llms](https://github.com/fbaratov/contractnli-llms).

Graph-structured retrieval-augmented generation (GraphRAG) has shown strong potential for improving multi-hop reasoning in large language models. However, existing approaches typically depend on dense entity–entity linking, which is computationally expensive, susceptible to hallucinated relations, and difficult to maintain under continual updates. We present Bipartite Graph Retrieval-Augmented Generation (BG-RAG), a lightweight bipartite framework that organizes knowledge into three layers, from chunks to entities to concepts, and replaces direct entity links with concept-mediated connections. This design enables more efficient graph construction, reduces reliance on hallucination-prone relation-edge prediction by avoiding explicit entity–entity relation extraction, and naturally supports incremental updates without global reconstruction. Experiments on HotpotQA, 2Wiki, and MuSiQue show that BG-RAG achieves competitive performance against RAG and GraphRAG baselines. Its concept-mediated structure also provides a lightweight alternative to dense entity–entity graph construction. These results suggest that BG-RAG is a lightweight alternative for retrieval-augmented generation, particularly when avoiding dense entity–entity linking is desirable.

Long-form narrative generation requires maintaining multi-dimensional consistency over extensive contexts, covering long-term constraints in character settings, event causality, spatial rules, and worldviews. Existing methods compress these heterogeneous elements into homogeneous representations, creating a topological mismatch that lowers retrieval signal-to-noise ratios and induces long-range logical collapse. We propose Topo-Narrator, a framework built on two complementary decouplings. Heterogeneous Topological Memory (HTM) organizes narrative state into a Character Graph, a Plot Chain, and an Environment Tree, providing topology-specific inductive biases. Plan-Reason-Write (PRW) multi-agent collaboration separates high-level reasoning, memory retrieval, and stylistic realization across dedicated agents. Experiments on MoPS and WritingPrompts show that Topo-Narrator consistently generates coherent narratives averaging over 13,000 words and substantially outperforms strong baselines on HANNA quality and fine-grained consistency metrics, validated by three frontier LLM judges and human evaluation.

Multi-agent debate is often motivated by the premise that multiple agents, modalities, and rounds of deliberation improve reasoning by combining complementary evidence and correcting individual errors. We test this premise in multimodal time-series analysis using **MAD-TS**, a prompting-based framework with modality-specific agents, cross-modal judging, and synthesis. Contrary to expectation, MAD-TS does not reliably outperform simpler baselines. Across 20 tasks in finance, weather, healthcare, and science, it is uniquely best on 2 of 23 outcomes and significantly beats a strong single-agent multimodal baseline on only one task, while requiring about 22× more tokens and 12× more wall-clock time. Adding modalities, increasing debate depth, or changing judge rounds can help some tasks but harm others. Trace analysis attributes 85% of errors to all three judges agreeing on the same wrong answer, a shared upstream bias that limits same-backbone multimodal debate as a general-purpose approach.

This tutorial provides a structured overview of code-switching for multilingual large language models (LLMs), connecting linguistic theory, NLP methodology, and human-centered perspectives. Code-switching is a natural communicative practice among multilingual speakers and is increasingly central to LLM interactions, where users bring mixed-language practices to prompts and models themselves may produce mixed-language outputs. The tutorial will cover theoretical foundations from sociolinguistics, code-switching behavior in human-AI interaction, datasets and evaluation benchmarks, synthetic data generation, multilingual LLM training and inference, mechanistic interpretability of unintended language switching, speech and multimodal code-switching, and societal impacts. By synthesizing work across NLP, linguistics, HCI, speech processing, and multilingual communities, the tutorial aims to give participants a coherent view of the field and identify open challenges for building multilingual LLMs that better reflect how multilingual users actually communicate.

With NLP research being rapidly productionized into real-world applications, it is important to be aware of and think through the consequences of our work. Such ethical considerations are important in both senior and junior roles of researchers and students (e.g. privacy, consent, fairness, among others). This tutorial will equip participants with basic guidelines for thinking deeply about ethical issues and review common considerations that recur in NLP research. The methodology is interactive and participatory, including discussion and group work related to case studies that involve bias and broader issues (e.g., environmental impact). Participants will gain practical experience on when to flag a paper for ethics review and how to write an ethical consideration section to be shared with the broader community. Most importantly, the participants will be co-creating the tutorial outcomes and extending tutorial materials to share as public outcomes.

As language models are increasingly optimized against the benchmarks used to evaluate them and deployed across diverse global populations, the assumptions behind evaluation are quietly breaking down. This tutorial focuses on a scientific approach for language model evaluation: a measurement-theoretic framework that treats LLM evaluation as a scientific measurement problem. We organize three hours around the classical structure of a measurement instrument and give NLP researchers usable tools for each part: (1) validity–when a benchmark score actually measures the capability it claims to, with emphasis on cross-cultural and cross-lingual transfer; (2) predictive measurement–inferring capability efficiently from sparse data via latent-variable models and cold-start prediction; and (3) design and incentives–why some benchmarks are gameable and what mechanisms improve robustness under explicit assumptions about evaluator access, developer objectives, and information leakage. The tutorial connects language model evaluation, latent-variable modeling, and mechanism design, three literatures that NLP researchers rarely encounter together, and equips the audience to build and interpret evaluations that remain trustworthy under deployment and strategic pressure.

LLM-based systems are increasingly deployed as language agents: long-running NLP systems that retrieve information, call tools, maintain memory, interact with users, and operate in changing environments. After deployment, these systems must contend with evolving user goals, updated knowledge sources, shifting APIs, new tasks, and previously unseen failure modes. Yet adaptation is still often handled through an ad hoc combination of prompt changes, periodic fine-tuning, retrieval updates, memory mechanisms, and manual intervention. The tutorial examines how post-deployment NLP systems can instead be designed as self-improving language agents, drawing on never-ending learning, lifelong learning, continual learning, and recent advances in agentic AI. We take a system-level view of learning: improvement may arise not only from changes to model parameters, but also from persistent changes to memory, knowledge, tools, policies, and other components of the agent and its harness. The tutorial combines literature review with practical running examples (code provided) illustrating different forms of post-deployment learning. We’ll examine empirical results and failure cases to understand when adaptation improves a system, when it doesn’t, and how such changes can be evaluated. We’ll cover a broader transition in NLP: from increasingly capable static models toward developing language systems that can continue to learn and improve responsibly throughout their deployed lifetime.

Despite well-known hallucination issues, LLMs have become an increasingly indispensable source of information, and the underlying technology has advanced rapidly to make their answers far more reliable. Early on, Retrieval-Augmented Generation (RAG) emerged as the dominant remedy, grounding LLM responses in external knowledge and evolving from simple retrieve-then-read pipelines into modular, graph-enhanced, and agentic systems. More recently, Agentic Deep Research has pushed the frontier further, equipping LLMs with autonomous planning, multi-hop investigation, and iterative synthesis to tackle open-ended questions that no single retrieval pass can answer. This tutorial offers an in-depth treatment of modern RAG and Deep Research, grounded in an AI-assisted systematic analysis of 2,000+ recent papers (2020–2026). Attendees will leave with a structured roadmap, evidence-backed practical recommendations, and a clear map of open research opportunities.

Diffusion language models (DLMs) have recently emerged as a non-autoregressive approach to language generation, enabling parallel, iterative, and refinement-based decoding beyond the conventional left-to-right paradigm. This tutorial offers the NLP community a technical map of this emerging field: starting from diffusion formulations for text, we clarify their connections to autoregressive language modeling, masked language modeling, and earlier iterative generation methods. We then survey recent progress in large-scale DLMs, efficient inference, training objectives, and post-training, including reinforcement learning for reasoning and alignment. Throughout the tutorial, we treat DLMs not as a solved replacement for autoregressive language models, but as a rapidly developing paradigm whose efficiency claims, controllability, practical assessment, and deployment implications require careful examination. We close by discussing representative open directions, from broader formulations such as flow-based language modeling to reliability, safety, and social bias as key requirements for the social acceptability of DLMs.

Long-form video editing over heterogeneous footage requires agents to coordinate source selection, multimodal analysis, timeline construction, narration and subtitle alignment, rendering, and revision while exposing intermediate state for inspection and repair. We present **Crayotter**, an open-source multimodal multi-agent demo system for prompt-driven long-form video editing. Crayotter organizes production around coverage-aware material preparation, artifact-grounded editing research, and tool-grounded timeline execution. Across these stages, retrieval reports, video analyses, editing blueprints, scheduler events, tool calls, intermediate renders, and final exports are treated as first-class artifacts rather than hidden transient state. The workbench supports local assets, agent-assisted retrieval, progress monitoring, artifact preview, failure diagnosis, interrupted-job resumption, and resource-aware asynchronous execution for long-running workflows. In a 23-theme evaluation, Crayotter achieves the highest human overall score (3.40/5) among the compared systems, with its largest margins in theme alignment, narrative coherence, and editing smoothness. These results show that long-horizon video editing agents can be made traceable, inspectable, and practically controllable through observable production artifacts.

Human evaluation is the gold standard for Natural Language Generation (NLG), but the tooling is fragmented: most platforms serve a single task or a single protocol, and none treat the growing practice of assisting annotators with an LLM as a measurable object of study. These gaps fall hardest on indigenous and under-resourced languages, where paid crowds are scarce, references are unavailable, and data governance matters. We present Aura, an open, self-hostable platform that unifies five evaluation protocols (Likert/DA, pairwise, ranking, error-span, post-editing) across eight task families under one segment schema and one result model, records writing-system and speaker provenance as first-class fields, and runs entirely offline. Its distinguishing feature is a grounded, advisory-only, logged LLM assistant paired with a built-in randomised experiment that estimates the effect of AI assistance on inter-annotator agreement and time-on-task. Aura’s agreement statistics match reference implementations to machine precision. We conducted a human evaluation experiment on three English→Indian-language ranking campaigns (N=9 annotators) which returned a mean usability rating of 4.06/5. Aura is released as open-source software.

Large language model-based generation pipelines commonly transform natural-language requests into structured plans before invoking expensive downstream generators, yet these plans are rarely verified before generation. We present Cinematic Compiler, an interactive system that compiles a short narrative brief into a typed Cinematic Intermediate Representation containing shots, design specifications, entity schedules, and explicit per-shot continuity constraints. A deterministic pre-generation pass checks that every shot carries the required character, costume, prop, and location facts, propagates missing constraints, and rejects contradictory briefs before any generation cost is incurred. At rendering time, the system conditions each shot on a shared character reference image using a frozen commodity image-generation backend, followed by vision-based auditing and reference-anchored repair. Across 15 benchmark briefs, the full constraint pass achieves 100% plan-level constraint coverage, compared with 2.2% for direct prompting, 22.8% for LLM-expanded prompting, and 36.5% for the intermediate representation without propagation. Reference anchoring improves identity-to-reference similarity from 0.605 to 0.714 across 13 briefs, with gains on 12 of 13 briefs, while matched text-only prompting produces no significant improvement. We further show that pairwise cross-shot similarity can reward degenerate near-static generations and therefore report identity preservation jointly with visual dynamic degree. The demonstration allows users to compile, inspect, edit, verify, generate, audit, and repair multi-shot stories through an interactive canvas, while supporting offline replay of stored runs.

Northeast India is home to over 220 indigenous languages, yet none are served by existing agentic AI systems. NE-Agent addresses this gap: a terminal-based agentic system that integrates the NE-Stack, a suite of purpose-built language AI components for Northeast India, into a unified conversational pipeline that dynamically selects between retrieval, translation, and transcription tools based on the user’s query. The system combines language identification across 11 languages (NE-LID), multilingual dense retrieval (NE-Embed, a LaBSE-based model fine-tuned for 10 Northeast Indian languages), Khasi-to-English translation (fine-tuned NLLB), speech transcription across eight languages (NE-ASR), and generation via a locally-hosted open-source LLM, requiring no API keys or cloud infrastructure. A lightweight LLM-based tool router dispatches each query to the appropriate tool, distinguishing NE-Agent from purely retrieval-augmented systems. NE-Agent is open-source, pip-installable (pip install ne-agent), and ships with a multilingual corpus spanning four languages. To our knowledge, it is the first publicly available agentic system built on indigenous Northeast Indian language AI infrastructure: https://github.com/MWirelabs/ne-agent.

We introduce HalluCiteChecker, a toolkit for detecting and verifying hallucinated citations in scientific papers. While AI assistant technologies have transformed the academic writing process, including citation recommendation, they have also led to the emergence of hallucinated citations that do not correspond to any existing work. Such citations not only undermine the credibility of scientific papers but also impose an additional burden on reviewers and authors, who must manually verify their validity during the review process. In this study, we formalize hallucinated citation detection as an NLP task and provide a corresponding toolkit as a practical foundation for addressing this problem. Our package is lightweight and can perform verification in seconds on a standard laptop. It can also be executed entirely offline and runs efficiently using only CPUs. We hope that HalluCiteChecker will help reduce reviewer workload and support organizers by enabling systematic pre-review and publication checks. Our code is released under the Apache 2.0 license on GitHub and is distributed as an installable package via PyPI. A demonstration video is available on YouTube.

Flagship language models appear saturated on benchmarks like MMLU (Hendrycks et al., 2021), scoring above 90% - yet benchmarks test only what the experimenter thought to ask, the availability bias of fixed question sets. LLMPEDIA makes this bias measurable and browsable. We recursively materialized ∼1.3M articles from three model families’ parametric memory (GPT-5-mini, DeepSeek-V3.2, Llama-3.3-70B) without retrieval, then audited a stratified sample of atomic claims against Wikipedia and a curated web stack, coloring every claim supported, refuted, or insufficient (Saeed and Razniewski, 2026). On a uniform random sample the true rate is 68.4% - more than 21 pp below MMLU - with 30.5% of claims insufficient: assertions no benchmark probes and the world’s largest encyclopedia cannot adjudicate - long-tail knowledge or plausible hallucination, the evidence cannot tell - extending to free text the coverage gap GPTKB established for triples (Hu et al., 2025). The resulting live, open encyclopedia lets visitors inspect this frontier one claim at a time through five one-click views - link-traversal exploration, claim-level factuality, cross-model and political-persona comparison, and a guided topic drill-down - each page, claim, and verdict at a stable URL. LLMPEDIA is live at https://llmpedia.net; see the demonstration video.

Evaluating open-ended generation remains non-trivial because quality varies across style, correctness, grounding, and safety. Moreover, a fixed prompt-response abstraction is insufficient for a reusable evaluation system as deployed applications increasingly expose multi-turn dialogues, retrieved evidence, and tool-calling traces. We present Pollux, a universal criteria-based evaluation framework centered on Pollux Judge, a compact Qwen3-4B-based LLM-as-a-Judge that evaluates such heterogeneous interactions through free-form user-defined criteria, scales, and output formats. Pollux follows the criteria methodology represented in the Pollux Benchmark (35 task groups, 2,115 expert-authored prompts), using it as a human-grounded resource for judge validation. The judge is trained with GRPO on Russian- and English-language data and is designed for bilingual deployment through a language-independent interaction schema. On the held-out expert-labeled test set, Pollux reaches a Spearman correlation of 0.744, exceeding every evaluated general-purpose baseline, including models with up to 1.1T parameters, while remaining suitable for local or OpenAI-compatible serving.

System demonstration video: https://drive.google.com/file/d/1PF6L-RcrS24JVUdaVRi01j237yTdftEd/

This paper proposes AI Tour Meeting, a group travel planning framework powered by multiple Large Language Model (LLM)-based agents. The agents are instantiated with distinct personas and collaboratively seek an itinerary that satisfies their constraints and preferences through natural language discussion.

The framework enables easy and flexible orchestration of such discussions by providing interfaces for configuring agent personas, discussion workflows, monitoring, and LLM deployment. Its primary use case is a simulation tool for analyzing the behavior of multiple LLM agents during tour planning discussions.

This paper demonstrates the utility of the framework by presenting system validation and several analytical results obtained using the framework.

The logit lens is a standard mechanistic interpretability tool for inspecting a Transformer’s predictions across layers. However, existing logit lens tools typically focus on individual checkpoints rather than interactive exploration of a model’s *training history*.

In this study, we introduce **LensLapse**, an open-source web application for interactive exploration of a model’s training history through token-level interpretability. Users can scrub a slider across public checkpoints from five model families and three multilingual checkpoint suites and observe how per-layer, per-position logit lens predictions evolve from frequency bias to factual knowledge.

For browser-compatible models, arbitrary user prompts are probed *live in the browser* via local inference over per-checkpoint model pairs. No server, no API key, no hosting cost, and prompts never leave the user’s device.

Moreover, **LensLapse** supports local models. A single ‘pip install‘ enables users to launch the application and explore their own models and training checkpoints without any additional web setup.

LensLapse is MIT-licensed and permanently hosted at iamtatsuki05.github.io/lenslapse.

- GitHub: https://github.com/iamtatsuki05/lenslapse - PyPI: https://pypi.org/project/lenslapse/ - docs: https://iamtatsuki05.github.io/lenslapse/docs/ - YouTube: https://youtu.be/OBnvG9Ru3j8

Large language models can transform project documents into software requirements, but direct generation may introduce unsupported claims and hidden inconsistencies. We present ReqLens, an interactive multi-agent NLP system that extracts candidate requirements from mixed project documents, links them to source evidence, exposes graph-based dependencies and conflicts, and generates a Software Requirements Specification (SRS) through a human-in-the-loop workflow. We evaluate ReqLens on 13 projects using direct-generation and modular baselines, as well as component ablations. ReqLens achieves the strongest overall balance across unsupported-requirement rejection, valid-requirement retention, and seeded-defect filtering. A formative study with four domain experts provided qualitative feedback on the system’s interface and review workflow. ReqLens is accompanied by code, data, an installable package, and a narrated screencast.

Final-answer accuracy hides where a mathematical solution first becomes inconsistent and whether a later correction merely masks that error. We present StateScope, an interactive system for detecting, localizing, and repairing solution-state drift: divergence between a solver’s evolving state and a symbolically verified reference trajectory. Under a controlled comparison, two solvers receive the same Computer Algebra System (CAS)-derived operation schedule and typed arguments. We provide two systems, 1) System C records the model’s textual state claim, 2) System D re-derives state in an external typed ledger. A SymPy oracle checks every transition and explains component-level differences without an LLM judge. Users can step through a run, edit an operation at any retained step, directly repair System C’s claimed state, continue the model, and repeatedly branch from earlier interventions. Our paired evaluation covers 250 problems and twelve open and proprietary models, and a human study measures diagnostic accuracy and time. StateScope targets reasoning researchers, model developers, and educators who need to inspect and manipulate more than a final answer.

Pragmatic understanding remains a fundamental challenge in natural language processing, particularly for underrepresented languages such as Cantonese. To alleviate the scarcity of resources in this area, we introduce YUE-PUB, the first large-scale pragmatic understanding benchmark for Cantonese. Constructed from the PUB dataset through automatic translation with human verification for quality assurance, YUE-PUB covers three core pragmatic phenomena: Implicature, Presupposition, and Reference. We conduct systematic experiments across multiple baseline models to evaluate their performance under several cross-lingual transfer settings, compare models with and without pragmatic topic awareness, and further analyze their sensitivity to different pragmatic phenomena as well as their Chain-of-Thought (CoT) reasoning behavior. Experimental results show that current large language models still exhibit substantial limitations in Cantonese pragmatic understanding, underscoring the value of YUE-PUB as an important resource for advancing pragmatic reasoning research in low-resource languages.

Recent studies have shown that code-switching data (CSD), in which multiple languages are mixed within the same context, can improve cross-lingual transfer and multilingual alignment in large language models (LLMs). Recent studies on synthetic code-switching for LLMs have primarily focused on bilingual transfer between English and a target language, leaving instruction-tuning settings that jointly mix three or more languages within each example largely unexplored. In this work, we investigate multilingual code-switching instruction tuning across four languages: English, Japanese, Korean, and Chinese. We evaluate multilingual understanding on Belebele and MMLU. Our experiments show that simple sentence-level multilingual CSD consistently improves the four-language average accuracy across all bilingual, trilingual, and quadrilingual settings for Gemma-2-2B, indicating that multilingual code-switching can be effective beyond bilingual transfer settings.

Corpus Query Language (CQL) supports precise corpus analysis but requires expertise in annotations, regular expressions, structural scopes, and platform-specific syntax. Natural-language interfaces simplify access, but their answers can be difficult to inspect, reproduce, or revise. We study Text-to-CQL as executable semantic parsing, mapping intents to inspectable queries. We introduce ECQL, a Sketch-Engine-oriented benchmark of 3,572 natural language–CQL pairs comprising 811 tutorial-derived and 2,761 synthetic examples, with a default split of 3,215 training and 357 test examples. ECQL focuses on Sketch Engine normalization, CQL Diagnostic Profile (CQL-DP) annotations, controlled semantic contrasts, audit-oriented metadata, and generalization protocols, alongside an evaluation suite covering syntactic validity, exact match, canonical match, and execution match. The full 357-example headline test set is independently double-annotated and adjudicated, with 96.9% agreement for intent clarity, 94.7% agreement for CQL–intent alignment, and a 3.1% post-adjudication intent-match error rate. Experiments with prompting, retrieval-augmented prompting, syntax repair, DeepSeek-R1 reasoning prompting, a GPT-4o Agent/API alternative, and supervised fine-tuning reveal persistent difficulties with corpus-query semantics. The best fine-tuned model reaches 85.6% execution accuracy on the random split, but drops to 68.9% on template-disjoint evaluation. ECQL provides a reproducible testbed for auditable semantic parsing over specialized linguistic query languages.

Although large language models (LLMs) show promise for generating educational feedback, it remains unclear how feedback should be designed (e.g., tone and coverage) to support answer revision and learner evaluations across learner profiles. We define six feedback designs for multiple-choice biology questions, including a baseline design and five variants with additional feedback elements, and conduct an empirical study with 321 high school students. We evaluate feedback using immediate revision performance and six subjective evaluation criteria, and analyze differences in subjective evaluations across learner profiles based on personality traits. Our results show that presenting task-relevant information clearly is associated with better immediate revision performance and is favorably evaluated across learner profiles, while we observe descriptive differences in evaluation patterns, particularly for informational novelty and affective framing. These findings support further investigation of personalized LLM feedback design.

We introduce Extended-SVD Low-Rank Adaptation, a parameter-efficient framework that extends the pretrained singular subspace while retaining a frozen spectral coordinate system and fixed singular-value-derived scaling. Its stabilized variant, PLORA (Procrustes Low-Rank Adaptation), applies localized Procrustes-inspired reorientation to the extended basis and confines task-specific learning to compact routing matrices P, R, and Q.

Across GLUE and vision benchmarks, PLORA achieves competitive performance with substantially fewer trainable parameters. In the full-network comparison, PLORA-r8 uses substantially fewer trainable parameters, although the performance gap varies by task. Beyond accuracy, PLORA produces task- and architecture-dependent changes in conditioning, effective rank, and subspace evolution.

Ablations show that Random Bases produce poorer routing-core conditioning and markedly lower top-r right-subspace overlap in the evaluated NLP tasks. In the evaluated settings, pretrained spectral bases are associated with more favorable optimization geometry, while Procrustes reorientation acts as an optimization-geometric regularizer. Overall, Extended-SVD provides a structured low-rank adaptation framework for studying the role of pretrained geometry in PEFT.

Retrieval-Augmented Generation (RAG) systems are increasingly used for biomedical Question Answering (QA). However, evaluating their effectiveness remains challenging due to the lack of reliable reference resources and the frequent misalignment between automatic evaluation metrics and human judgments. One practical question is how evaluation results should be used to guide development decisions, especially when they correlate weakly with human judgments and may even lead to different conclusions. We explore how to use a small amount of human preference signals to calibrate direct assessment evaluation metrics, and show that calibrated metrics consistently outperform baselines that either treat all metrics equally or rely on a single best metric across multiple datasets.

Large Language Models often generate factually correct responses that fail to maintain user-requested perspectives, blending multiple perspectives and forcing users to filter through unwanted information—a perspective control failure that factual accuracy metrics cannot detect. We introduce PerspectiveBench, a benchmark quantifying perspective alignment by constructing scenarios where multiple valid reasoning paths coexist, and propose the Perspective Alignment Score (PAS) to measure adherence to specified perspectives. Experiments reveal that state-of-the-art models successfully include target content but fail to exclude non-requested perspectives. Providing documents through RAG improves factual grounding but does not resolve perspective drift, as models prioritize comprehensive coverage over strict user constraints. Code and data are available at https://github.com/sunghoon014/PerspectiveBench.

Post-training quantization methods such as GPTQ and AWQ minimize reconstruction error without asking which weights matter for the task. We combine six interpretability signals to select a sparse skeleton of task-critical MLP neurons that is kept at 16-bit while the rest of the model is quantized. On Gemma-2-2B fine-tuned for SQL generation, the 0.54% skeleton with 3-bit MLPs and 8-bit attention retains 100% of baseline-correct outputs at 4.03x compression, versus 75.2% for uniform 4-bit. On Llama-3-8B fine-tuned for Cypher generation, where uniform 4-bit loses 23 points, the 0.46% skeleton recovers 14.4 of them and a random skeleton of equal size recovers none. Three signals are individually effective and three are not, and the effective set differs between models, so the consensus rule provides robustness no single signal does. Selection matters where uniform quantization leaves headroom, which suggests measuring headroom, selecting by consensus, and protecting as a recipe for larger models. GPTQ error compensation degrades 3-bit layouts by 16 to 18 points regardless of protection.

There are inconsistent conclusions about the impact of calibration data on Post-Training Quantization (PTQ) across studies, specifically regarding the factors that make the method effective, such as language that we explore in this work. We investigate this question by evaluating GPTQ and its mixed-precision variants across three low-bit settings, using language-targeted calibration sets across five language families. By evaluating performance across both perplexity and downstream tasks, we find that selecting an appropriate calibration language is generally beneficial, though model sensitivity varies with bit-width, target language, and task difficulty. We further show that mixed-precision techniques expand the representational capacity of low-bit models, occasionally allowing 2-bit models to approach 3-bit performance when paired with language-targeted calibration, while increased effective-bits remains small. Furthermore, we explore the correlation between these factors and observe that the calibration set sensitivity is influenced by different combinations of factors for each controlled variable, which can be explained by the downstream task results. Finally, while the benefit of language-targeted calibration transfers to sibling languages within the same family, the magnitude of this transfer does not reliably predicted by their closeness. Together, these findings show calibration data selection as a deliberate design choice in multilingual PTQ rather than an incidental one. Our code and result data can be accessed in https://github.com/rafifpriyo/multilingual-calibration-study.

Interpretable linguistic features offer a promising approach for explaining *why* a given text appears machine-generated, particularly for non-expert users. However, existing findings on which features reliably indicate LLM-generated text remain fragmented across feature sets, models, and text domains. To address this gap, we conduct a large-scale empirical study assessing the robustness of linguistic signals for characterizing AI-generated text. Our analysis covers 284 interpretable linguistic features across outputs from 30 LLMs and 10 text domains under cross-model and cross-domain generalization settings. We show that classifiers based solely on linguistic features can reliably distinguish AI-generated from human-written text. However, many previously proposed indicators prove strongly context-dependent, with the exception of measures of lexical richness, which remain robust signals across model families and text domains. These results demonstrate which linguistic signals generalize across contexts and provide a foundation for more reliable, interpretable analyses of AI-generated language.

Rubric-based evaluation is a promising approach for assessing open-ended outputs from LLM-based research agents, particularly in paper reproduction, where direct paper-to-repository comparison is prone to hallucinations. However, constructing paper-specific rubrics requires substantial expert effort, limiting the scalability of benchmarks such as PaperBench. In this work, we present, to our knowledge, the first systematic meta-evaluation of LLM-generated rubrics for paper reproduction. We reformulate rubrics into a checklist-style format and evaluate four generation settings across two backbone models. We meta-evaluate generated rubrics intrinsically by semantic similarity and extrinsically by score alignment with ground-truth rubrics. Our results show that the augmented settings substantially improves downstream evaluation alignment, with the strongest setting approaching the human baseline, while intrinsic gains are more modest. Further analyses reveal that LLM-generated rubrics are often overly fine-grained, biased toward high scores, and less adaptive to paper domains, highlighting both the promise and limitations.

Language models frequently generate outputs in unintended languages or scripts—a phenomenon known as off-target generation. While existing research has focused on language selection, the dimension of script knowledge remains understudied: before any linguistic understanding can occur, users must recognize the graphic symbols in a model’s response. We investigate whether Small and Large Language Models (SLMs and LLMs) possess script knowledge by testing them on multi-scriptic languages. Through two complementary experiments, we evaluate whether models (1) adapt their output script to match the input, and (2) follow explicit instructions to generate text in a specified script. The models we tested demonstrate substantial script knowledge: they all achieve a near-perfect Latin script fidelity (more than 98%), and follow script instructions with high frequency. Nevertheless, we notice differences between LLMs and SLMs, with higher scores for LLMs including for non-standard script combinations.

Graphical User Interface (GUI) agents powered by Multimodal Large Language Models (MLLMs) promise human-like interaction with software applications, yet long-horizon tasks remain challenging due to memory limitations. Existing approaches either truncate history or rely on simple textual summaries, which risk losing critical information when past visual details become necessary for future decisions. In this paper, we propose PAL-UI (Planning with Active Look-back), a novel framework that enables GUI agents to adaptively retrieve past observations when required. PAL-UI combines a dual-level summary agent, capturing both observation-level cues and action-level outcomes, with a dedicated retrieval tool that allows the agent to recall specific historical screenshots during planning. We curate a step-level instruction dataset of 8.6K samples from mobile GUI navigation trajectories and train PAL-UI-3B and PAL-UI-7B models based on Qwen2.5-VL. Extensive experiments show that PAL-UI significantly outperforms baseline models and prior methods in mobile GUI tasks, even under data-efficient settings. Moreover, PAL-UI exhibits strong cross-domain generalization, achieving notable improvements in web navigation without additional training. Our work highlights the potential of active memory retrieval for long-horizon planning capabilities of vision-based GUI agents.

Sinhala Sandhi splitting recovers the constituent words or morphemes hidden inside a phonologically merged surface form. The task is important for Sinhala NLP because Sandhi obscures lexical boundaries, but no prior published work has established a neural benchmark for Sinhala Sandhi splitting. We present a character-level sequence-to-sequence study based on SandhiLex, using native Sinhala Unicode input and evaluating recurrent encoder-decoder models for affixational and more complex lexicalized, derivational, and etymological Sandhi. A bidirectional LSTM encoder with a unidirectional LSTM decoder yields the best results on the hard subset, achieving 68.40% exact-match accuracy and 82.08% character-level accuracy. On the full affixational subset, the same model achieves 94.00% exact-match accuracy. Ablations show that bidirectional encoding is the largest contributor to performance, while native Sinhala script improves exact match accuracy over romanized input. Qualitative analysis indicates that many errors are near misses involving boundary adjacent characters or plausible but incorrect phonological substitutions. These results establish an empirical baseline for Sinhala Sandhi splitting and identify data scale, Sandhi type conditioning, and attention-based decoding as the main directions for future work.

Contrastive vision-language models have achieved remarkable progress through large-scale pretraining. Recent work has shown that removing English-only caption filters and pretraining on global data is effective for improving multicultural performance. We study whether such global pretraining is sufficient for culture-specific understanding, or whether further adaptation with natively sourced data can boost performance beyond what global pretraining alone achieves. To enable this investigation, we present WAON, the largest publicly available native Japanese image-text dataset constructed from native Japanese web content in Common Crawl, containing approximately 155 million examples. We also introduce WAON-Bench, a manually curated Japanese cultural benchmark spanning 374 classes. Through comparative fine-tuning experiments on multiple Japanese image-text datasets, we observe that models fine-tuned on WAON consistently achieve stronger performance on Japanese cultural benchmarks than those fine-tuned on English-to-Japanese translated data. Controlled experiments at matched scale, filtering, and training budget across two model families further indicate that native web origin is the primary driver of this gain. We release our dataset, benchmark, model, and code.

Retrieval-augmented generation is widely used in professional writing, yet whether retrieval grounds revision or merely injects templates is rarely tested where "correct" has a definable meaning. Patent claim amendment supplies that signal: the examiner names the attacked limitation and cites prior art, providing per-case ground truth. We release three artifacts: (i) a corpus of 7,385 USPTO prosecution cases with XML-aligned pre/post claims, rejection, and cited prior art; (ii) a seven-probe battery comparing random and structural-match retrieval as two policies under a fixed prompt scaffold; (iii) a deterministic five-channel metric (C1-C3 and C5 in main, C4 supplementary) requiring no LLM evaluation. Across 9,600 pre-registered calls on four frontier LLMs (Claude Sonnet 4, Claude Haiku 4.5, GPT-5.4, GPT-4o-mini), no tested model exhibits detectable classical prior-injection behavior; retrieval effects are small and direction-inconsistent between random and structural retrieval, and the null is unchanged under a dense (semantic) retriever, across retrieval depths k in 1,3,5,10, and under a paraphrase-sensitive grounding metric. Revision locality reveals a model-specific difference that the template channel misses. The four-cell taxonomy, which we treat as exploratory, leaves the prior-injector cell unoccupied.

Prompting has transformed large language models into general-purpose decision systems, where token-level probabilities are often treated as indicators of model preference and correctness. However, directly interpreting these probabilities is challenging due to calibration disparities, tokenizer variability, and inconsistent output distributions across models, limiting their statistical comparability. This raises a fundamental question: *when can a model be said to be confident in its decision or preference?* Addressing this question remains an open and important area of research. We introduce **PLAUSIEVE**, a novel distribution-based statistical framework that estimates model preference through a forward pass without token generation. **PLAUSIEVE** provides a calibration-robust and model-independent mechanism for quantifying preference without relying on fixed thresholds or individual tokens. We apply **PLAUSIEVE** in estimating model belief, across two prompt settings (direct and epistemic), two confidence levels, and five models from four families to study prompt–architecture interactions. Our analysis shows significant improvements under epistemic prompting, with nearly 100% gains in certain model combinations. Notably, **Llama-3.2-3B** outperforms the larger **Gemma-7B** under epistemic prompting in terms of both decisiveness and veracity, demonstrating that model architecture plays a more significant role than scale in shaping belief formation.

This work investigates whether Chain-of-Thought (CoT) prompting benefits patent claim generation. We demonstrate a counter-intuitive but important empirical finding: implicit CoT, where reasoning is kept internal rather than explicitly verbalized, consistently outperforms explicit CoT. Through systematic analysis, we show that explicit CoT can introduce an unnecessary information bottleneck for claim generation. Verbalized reasoning may compromise the quality of final outputs through three specific mechanisms: abstraction of critical details, disruption of internalized generation patterns, and cascading error propagation. Our findings provide new insights into structured text generation and CoT applications.

When LLMs annotate subjective content such as predicates of personal taste (PPTs), e.g., "Eliza tasted the cookie. It tasted delicious", a consequential but underexplored question is whose perspective they adopt: the narrator’s (default) or Eliza’s (shift toward the character)? Humans tend to attribute "delicious" to Eliza because tasting implies first-hand experience. We introduce GRANI (Graded Response Analysis for Narrative Implications), a protocol for evaluating LLMs’ graded understanding of such phenomenon. Applying GRANI to PPTs, we test whether text-trained LLMs reproduce human perspective shifts across sensory modalities in English and Chinese. We found that instruction-tuned models partially capture the human pattern, in which taste yields the strongest shift toward the character, vision the weakest, and smell an intermediate effect, while reasoning-distilled models are less consistent. Moreover, some models encode modality distinctions internally even when their behavioral responses do not. These findings suggest that subjectivity inference can emerge from text exposure alone, while highlighting a reliability concern: LLMs may not consistently adopt the intended perspective-holder when evaluating subjective content.

Synthetic supervision can bootstrap semantic parsers in new search domains, but prompt-induced diversity is useful only when generations are faithful to the structured intent. We explore this for cold-start e-commerce search: 118,098 synthetic pairs produced by 6 LLMs with 9 persona-conditioned prompt families on a 47-field UK automotive schema, against 1,214 human-annotated and 15,304 production queries. We show that the IPIP personality bundle improves parser F1 in all four matched contrasts (Holm-corrected p < .001), and all three bundles have a positive main effect over the complete 2^3 grid. Syntax conditioning is the axis that moves generations towards real traffic. It also induces lexical diversity. Alignment sKL, the divergence between von Mises-Fisher distributions fitted to synthetic and human-annotated queries, is an unlabelled proxy for parser F1 (rho = -0.917 on intent-elicited human queries, lower is closer). An LLM committee, checked against a blinded author pass, triages the 927 queries missed by every parser and labels 37% as hallucinated. Dropped constraints dominate. The four syntax-bearing families have the largest hallucinated shares, 38 to 42%, while IPIP alone has the lowest at 28%.

Chain-of-thought (CoT) prompting improves performance on complex reasoning tasks, but often triggers unnecessary computation on simpler queries. A natural approach is to use the model’s own confidence as a training-free signal to gate this decision, answering directly when confident and invoking CoT only when uncertain. Across four training-free confidence signals, three model families, and seven reasoning benchmarks, we find that the effectiveness of confidence-gated CoT varies, and it is unclear how to predict whether it will work for a new model. We address this with two key contributions. First, we identify that gating fails in the following circumstances: signal saturation, where confidence scores collapse to a narrow range and lose discriminative power; and selection bias, when confidence-gated routing sends the longest queries to CoT. Second, we propose a novel Pareto-optimal threshold selection method. This has the dual ability to preserve accuracy while reducing costs where the confidence signal is strong, and maintains accuracy where it is weak. Where the confidence signal is strong, gating reduces CoT usage by 30% without sacrificing accuracy. These findings provide practical guidance for evaluating and deploying models that selectively use CoT.

Multimodal sentence embedding models typically leverage image-caption pairs in addition to textual data during training. However, such pairs often contain noise, including redundant or irrelevant information on either the image or caption side. To mitigate this issue, we propose MCSEO, a method that enhances multimodal sentence embeddings by incorporating fine-grained object-phrase alignment alongside traditional image-caption alignment. Specifically, MCSEO utilizes existing detection and segmentation models to extract accurate object-phrase pairs, which are then used to optimize a proposed contrastive learning objective tailored to object-phrase correspondence. Experimental results on semantic textual similarity (STS) tasks across different backbone models demonstrate that MCSEO outperforms image-caption-only baselines on average, and that it is complementary to existing methods: combining MCSEO with KDMCSE yields further gains. A random-pairing control and a manual evaluation of the extracted pairs additionally indicate that the improvements stem from correct object-phrase alignment.

Multimodal fake news detection requires models to reason over both textual and visual evidence while remaining robust to distribution shifts in social media graphs. Existing approaches often concatenate modality-specific representations or rely on post-hoc fusion, which can leave image and text features in misaligned semantic spaces. Graph-based methods further introduce structural bias: node popularity, degree, or connectivity may become shortcut signals that correlate with labels in the training distribution but do not causally determine veracity. We present CIL, a causal-intervention and LLM-synergized framework for multimodal fake news detection. CIL first uses CLIP to obtain aligned image and text representations and constructs a multi-view semantic graph from cross-modal, intra-modal, and fused similarities. It then uses a large language model to produce high-confidence pseudo labels, which are injected as weak supervisory anchors. Finally, CIL mitigates degree-induced structural confounding with inverse propensity weighting and enforces counterfactual consistency under edge-drop interventions. Experiments on Twitter, PHEME, and Weibo suggest that each component contributes to performance, with the largest gains coming from aligned multimodal representations and causal debiasing.

Automated Essay Scoring (AES) for English proficiency assessment increasingly relies on pretrained transformer models, yet these models are typically trained on general-domain English and may under-represent second-language learner writing. This study investigates whether domain-adaptive continued pretraining (DAPT) on a learner-writing corpus improves transformer-based AES for English proficiency assessment. We perform DAPT on BERT, RoBERTa, and DistilBERT using the EFCAMDAT corpus, then compare the adapted models with their original checkpoints on two English proficiency test datasets, FCE and IELTS, in both in-domain scoring and few-shot cross-dataset transfer. Full-corpus DAPT produces mixed effects across models, datasets, and metrics. Subsequent lexical and syntactic analyses suggest mismatches between EFCAMDAT and the downstream datasets in proficiency level, genre, and communicative purpose. We therefore repeat DAPT using proficiency-specific EFCAMDAT subsets across all three encoder architectures. Proficiency-specific DAPT frequently outperforms full-corpus DAPT and, in some settings, even the non-adapted baseline. Overall, continued pretraining on learner writing can improve in-domain AES, but its benefits depend on both the proficiency composition of the pretraining data and the underlying encoder architecture, and do not consistently extend to cross-test transfer.

Evaluating conversational therapist competence in Motivational Interviewing (MI) requires multi-session assessment, yet current benchmarks rely on simulated patients that fail to model progression from session to session. We present MuSe-MI (Multiple Sessions Motivational Interviewing), a simulated patient framework grounded in Motivational Enhancement Therapy (MET), a multi-session clinical protocol of MI. MuSe-MI regulates dialogue generation through four evolving internal states from the MET clinical manual (anger, readiness to change, problem recognition, and self-efficacy), paired with an inter-session life event simulator and structured memory. Blind evaluations by licensed clinical psychologists confirm that patient dialogue reflects these internal states with high inter-rater agreement. Experiments show that MuSe-MI reliably distinguishes conversational therapist MI adherence levels based on patient language progression, establishing an automated multi-session evaluation benchmark.

While large language models (LLMs) have achieved remarkable linguistic capabilities, a profound question lingers at their core: do these models truly comprehend context or simply excel at pattern matching on an unprecedented scale? Contextual understanding in LLMs refers to the ability to correctly extract relevant information from a given context, integrate it into a coherent internal representation, and reason over it to produce factually consistent and contextually grounded responses. However, traditional methods such as BiLingual Evaluation Understudy (BLEU) and perplexity simply measure surface-level performance. This reveals a critical gap in question answering (QA), where responses must be contextually grounded rather than simply being memorized associations. To fill this void, we propose a novel knowledge graph (KG) based evaluation framework for LLM contextual understanding in QA. Central to this is Semantic Structural Similarity for KGs (S3KG), a hybrid similarity measure combining structural and semantic signals into a single score. In addition, a diagnostic analysis framework is developed to identify and categorize reasoning errors at the triplet level, enabling fine-grained analysis of model failures. Together, across nine benchmarks, S3KG achieves F1 gains of up to +7.6 points over the strongest baseline and AUROC up to 0.973.

Large Language Models (LLMs) are powerful tools for answering user queries, yet they remain highly vulnerable to jailbreak attacks. Existing guardrail methods typically rely on internal features or textual responses to detect malicious queries, which either introduce substantial latency or suffer from randomness in text generation. To overcome these limitations, we propose SelfGrader, a lightweight guardrail method that formulates jailbreak detection as a numerical grading problem using anchored token-level logits. Specifically, SelfGrader evaluates the safety of a user query within a compact set of numerical tokens (NTs) (e.g., 0–9) and interprets their logit distribution as an internal safety signal. To align these signals with the target safety rubric, SelfGrader constructs Probably Approximately Correct-guided ICL anchor examples and introduces a dual-perspective scoring rule that considers both the maliciousness and benignness of the query, yielding a stable and interpretable score that reflects harmfulness and reduces the false positive rate simultaneously. Extensive experiments across diverse jailbreak benchmarks, adaptive attacks, benign prompt benchmarks, multiple LLMs, and state-of-the-art guardrail baselines demonstrate that SelfGrader achieves strong robustness with low false positive rates, memory overhead, and latency. Warning: Some content generated by LLMs may be offensive.

Selective retrieval aims to make retrieval-augmented generation (RAG) more efficient and reliable by skipping retrieval when an LLM’s parametric knowledge suffices. Despite promising results, existing methods are constrained by a binary design choice: either retrieve from a single external source or skip retrieval and let the LLM directly produce the final answer. We argue that this fallback underestimates the model’s knowledge and obscures the more general multi-source decision problem that arises in practical systems. We propose Self-Routing RAG (SR-RAG), which casts selective retrieval as knowledge source selection and treats the LLM itself as a first-class knowledge source. SR-RAG learns to select an appropriate knowledge source, optionally verbalize parametric knowledge, and answer using the selected source, all within a single left-to-right generation pass. SR-RAG further augments source selection by combining LLM-based uncertainty with a flexible external policy datastore to improve decision calibration. Across four benchmarks and three 7B-class LLMs, SR-RAG outperforms a strong selective retrieval baseline by 8.5%/2.1%/4.7% while performing 26%/40%/21% fewer retrievals, and it achieves favorable accuracy-latency trade-offs without dataset-specific threshold tuning.

Manchu is a typical low-resource agglutinative language, for which existing digital resources are largely limited to character recognition and basic annotation, while sentence-level generation remains underexplored. In this paper, we study constrained sentence-level Manchu continuation. We construct the first context-target corpus for this continuation task, consisting of 3214 examples collected from Manchu Stories in Translation, Common Manchu Dialogues, and other sources, covering both narrative and conversational genres. To improve robustness to rich suffixation, relatively flexible word order, and frequent OCR or transcription noise in Manchu, we propose a Multi-Granularity Data Augmentation (MGDA) strategy that expands training data through character-level noise injection, phrase rearrangement, and suffix perturbation, without relying on external dictionaries or translation models. Experiments on ByT5-small and Qwen2-1.5B show that MGDA substantially improves Manchu text generation performance: BLEU-4/ROUGE-L increases from 2.03/2.59 to 17.60/42.70 for ByT5, and from 1.98/6.42 to 14.12/41.49 for Qwen. Further ablation studies demonstrate the complementarity of the three augmentation granularities and suggest potential benefits of byte-level modeling in this experimental setting.

Knowledge Graph Question Answering (KGQA) is particularly challenging for Persian, a complex and low-resource language with significant ambiguity. To address this, we introduce PeMA-KGQA (Persian Multi-Agent Knowledge Graph Question Answering), the first multi-agent LLM framework for Persian KGQA, arguing that an agent-based approach is essential for robust reasoning. Our system operates on FarsBase and is evaluated on both the ParsSimpleQA (one-hop) and PeCoQ (compositional) benchmarks. The framework employs a plan-and-execute paradigm where a dedicated Planner agent first decomposes a question into a structured executable plan using a toolbox of operators. Subsequent specialized agents—an Entity Grounder, a Relation Selector, and a Path Scorer—then execute this plan, effectively grounding mentions, selecting relations, and scoring reasoning paths to produce the final answer. This multi-agent decomposition provides a transparent and interpretable reasoning trace. PeMA-KGQA sets a new state-of-the-art on ParsSimpleQA with 21.14% improvement in Hits@5 and achieves 80.67% in Hits@5 on the complex multi-hop questions of PeCoQ.

Large language models pre-trained on code (Code-LLMs) have achieved remarkable performance on coding tasks. Despite their success of processing programming language, their application to natural language tasks has largely been limited to narrow tasks and ad-hoc model selection. In this paper, we present Code-Prompt, a comprehensive prompt-based framework to benchmark Code-LLMs over ten NLP tasks. We evaluate Code-Prompt on 13 Code-LLMs using code-aligned prompts and compare them with their parallel natural language LLMs (NL-LLMs) using natural language prompts.

Our results show that Code-LLMs perform on par with NL-LLMs across a range of NLP tasks, while producing more format-consistent generation with less redundancy. Our cross-lingual experiments further indicate that knowledge in Code-LLMs is well shared among different programming languages, whereas performance across different human languages still varies significantly. All our code and data will be made publicly available at https://github.com/chen-yuxuan/code-prompt.

Time and event expression extraction are fundamental temporal reasoning tasks, but the problem remains difficult due to annotation ambiguity, domain sensitivity, and unstable model behavior. Existing evaluations focus on in-domain performance, offering limited insight into reliability under distribution shifts. We evaluate multiple model configurations across families, architectures, and reasoning strategies over four dimensions of generalization, examining transfer from base performance, cross-dimensional correlations, and the effects of scale, architecture, and prompting. This provides a systematic study of how prompted LLMs generalize in time and event expression extraction tasks. We find that strong base-task performance generally predicts better generalization. However, this relationship weakens under substantial distribution shifts. Inductive prompting performs most consistently across domain shift, adversarial perturbations, compositionality, and length increase, while gains from scale, architecture, and deductive and abductive prompting strategies are uneven and dimension-specific. We conclude that LLM generalization in temporal extraction tasks cannot be predicted from any single dimension alone and cannot be reliably inferred from in-domain or single-dimension evaluations, highlighting the need for reasoning strategies that generalize across dimensions.

Similarly to humans, LLMs are susceptible to the anchoring effect in price negotiations, which can lead to unfavorable or unintended outcomes. To ensure LLM safety in dialogue tasks such as negotiation, a mechanistic understanding and mitigation methods are essential. We propose four variants of a method to identify Anchoring Heads, which are the attention heads responsible for the anchoring effect, while preserving those essential for general tasks. Experimental results show that suppressing these heads mitigates the anchoring effect by over 16%, achieving complete mitigation in certain models without compromising general performance. Furthermore, the observation that suppressing random heads fails to mitigate the anchoring effect suggests that specific attention heads drive the anchoring effect by paying excessive attention to the anchor.

Topic models summarize large text corpora, but top-ranked words often provide only a limited representation of topic semantics. Sparse autoencoders (SAEs) offer a way to move beyond word-level descriptors by extracting interpretable features from dense representations, yet how feature interpretability relates to topic-inference quality remains unclear. We introduce MonoTM, an interpretable topic modeling framework that decouples these roles. Across three benchmark corpora, we show that document–topic mixture estimation and semantic interpretation favor different SAE configurations and feature subsets. MonoTM estimates mixtures from the full SAE bag-of-features representation and, with them fixed, learns topic descriptors over a separate vocabulary of corpus-grounded semantic features. This design preserves global topic structure while representing topics with semantic units more meaningful than individual words, making them more useful for downstream corpus analysis.

Tool use, such as web search, has become a standard capability even in freely available large language models (LLMs). However, existing benchmarks evaluate temporal reasoning mainly in static, non-tool-using settings, which poorly reflect how LLMs perform temporal reasoning in practice. We introduce Time Puzzles, a constraint-based date inference task for evaluating iterative temporal reasoning with tools. Each puzzle combines factual temporal anchors with (cross-cultural) calendar relations and may admit one or multiple valid dates. The puzzles are algorithmically generated, enabling controlled and continual evaluation. Across 13 LLMs, even the best model (GPT-5) achieves only 55.3% accuracy without tools, despite using easily searchable facts. While web search improves performance, models perform substantially better when constraints are rewritten with explicit dates, removing the need for factual lookup. These results reveal a gap in reliable tool use for iterative temporal reasoning.

While the field of vision-language (VL) has achieved remarkable success in integrating visual and textual information across multiple languages and domains, there is no dedicated framework for assessing human-centric alignment in vision-language systems.

We offer two contributions to address this gap. First, we introduce Anthropogenic Regional Adaptation: a regional-adaptation framework that aims to optimize model relevance to specific regional contexts while ensuring the retention of global generalization capabilities. Second, we present a simple, but effective adaptation method named Geographical-generalization-made-easy (GG-EZ), which utilizes regional data filtering and model merging.

Through comprehensive ablations on 3 VL architectures: large vision-language models (LVLM), text-to-image diffusion models, and vision-language embedding models, and a case study in Southeast Asia (SEA) regional adaptation, we demonstrate the importance of Anthropogenic Regional Adaptation and GG-EZ, showing 5-15% gains in cultural relevance metrics across SEA while maintaining over 98% of global performance and even occasionally surpassing it.

Our findings establish Anthropogenic Regional Alignment as a foundational paradigm towards applicability of multimodal VL architectures in diverse regions and demonstrate a simple-yet-effective baseline method that optimizes regional value alignment while preserving global generalization.

Large language models (LLMs) can achieve strong performance on discourse relation prediction through fine-tuning. Yet it remains unclear whether such performance reflects a robust modelling of inter-sentential relations or a reliance on shallow lexical cues. This paper presents a behavioural audit of fine-tuned models across explicit and implicit settings. In explicit discourse relation prediction, we find that fine-tuned models rely heavily on connectives as categorical triggers, rather than integrating argument semantics. More surprisingly, in implicit discourse settings, we observe a severe performance degradation when masking semantically light function words from the arguments, suggesting that models may also exploit subtle surface-level cues beyond explicit connectives. These results suggest that high accuracy alone is insufficient to demonstrate robust discourse processing. To address this limitation, we propose a causally validated diagnostic framework that measures whether model predictions are grounded in linguistically meaningful evidence. Our metric provides a complementary perspective to standard accuracy-based evaluation and establishes a rigorous auditing toolkit for future work on discourse relation prediction.

LLM safety guardrails often behave unevenly in localized multilingual settings, especially with code-mixed vernaculars and non-Latin scripts. We introduce RabakBench, a Singapore-localized multilingual safety benchmark for Singlish, Chinese, Malay, and Tamil, built through a semi-automated, human-validated Generate–Label–Translate pipeline. The benchmark contains 1,341 unique examples in four language versions, yielding 5,364 entries across six harm categories, with LLM-assisted labels validated against human consensus. Evaluating 13 commercial and open-source guardrails reveals substantial cross-language and system-level variation, highlighting localized and script-sensitive safety gaps in the Singaporean context. RabakBench provides both a diagnostic resource and a reproducible, human-in-the-loop framework for constructing localized safety benchmarks.

Large language models (LLMs) achieve strong performance across many tasks, but their high computational cost limits deployment in resource-constrained environments. Knowledge Distillation (KD) offers a practical solution by transferring knowledge from a teacher model of a larger size to a smaller student model. While prior work has mainly examined task-specific or small-scale settings, the post-training stage for building general instruction-following models has received limited attention. In this paper, we conduct a systematic study of KD in post-training using the large-scale Tulu 3 dataset. We find that KD outperforms supervised fine-tuning (SFT) in low-data regimes, but its advantage diminishes as more training data is added. Distilling from a stronger instruction-tuned teacher restores substantial gains even with abundant data, indicating that KD remains effective when the teacher provides knowledge that the student cannot easily acquire from the training data alone. We further study domain-specific, low-resource scenarios and propose a two-stage KD strategy that leverages synthetic teacher-labeled data followed by refinement on human annotations. This method consistently improves student performance, providing practical guidance for building compact models in data-scarce environments.

This study proposes a new LLM-as-a-judge method for relative evaluation of LLM responses based on both required criteria (i.e., minimal conditions that responses must satisfy) and bonus criteria (i.e., desirable additional qualities). Recent LLM-as-a-judge methods have improved the stability and accuracy of relative evaluation through the use of automatically generated criteria. However, existing methods rely solely on the required criteria, which may limit their ability to distinguish closely matched responses. To address this issue, the proposed method first evaluates responses based on required criteria and, in the case of a tie, further evaluates them using bonus criteria, enabling more fine-grained comparisons. Experiments on the Writing, Roleplay, and Humanities categories of the MT-Bench dataset demonstrate that the proposed method achieves higher agreement with human judgments than existing methods.

Singlish is a creole rooted in Singapore’s multilingual environment that continues to evolve alongside social and technological change. We examine diachronic stylistic change across a decade of informal digital messages and ask whether Large Language Models (LLMs) can generate temporally neutral outputs approximating the stable characteristics of the variety. Using lexical, pragmatic, psycholinguistic, and encoder-based features, we find that stylistic separability increases with temporal distance, driven primarily by structural features such as length and complexity. Evaluated against a null distribution baseline, most LLMs fail to achieve both authenticity and temporal neutrality simultaneously at their most authentic prompting configuration. This reveals a structural trade-off: models generating realistic Singlish tend to inherit its temporal biases, while temporally neutral models tend to produce inauthentic outputs. These findings position temporal neutrality as a diagnostic metric for assessing sociolectal grounding in LLMs.

Political debates shape public opinion, but many readers increasingly encounter debate takeaways through AI-generated summaries and commentary rather than full broadcasts. This paper studies whether fine-grained lexical choices in debate transcripts can influence the evaluative framing of GPT-generated political commentary. We introduce controlled word-level interventions by replacing embedding-identified key lexical items with near-synonymous alternatives, allowing us to examine how minimally invasive lexical changes affect model-generated interpretation. Across U.S. presidential debate segments, we find that single-word substitutions can shift the stance of GPT-generated commentary, but these effects are probabilistic, context-dependent, and sensitive to the speaker’s communicative goal. Substitutions are more likely to produce favorable stance shifts in critical contexts than in promotional ones, suggesting that GPT-generated commentary may amplify adversarial framing more readily than self-promotional framing. We further analyze which linguistic cue types are associated with substitution success, including sentiment direction, part of speech, and contextual suitability, and show that sentiment alone is insufficient to explain model responses. Finally, a small exploratory human evaluation suggests partial alignment between model-based stance shifts and reader preferences in promotional cases, while also revealing substantial subjectivity in human judgments. These findings highlight how candidates’ word choices may be transformed when political discourse is filtered through GPT-generated commentary, and caution against treating model-internal stance metrics as direct substitutes for large-scale human audience evaluation.

Modern visual CAPTCHA has evolved from simple perception tests into real-world spatial reasoning challenges. It remains unclear whether state-of-the-art vision-language models can solve such tasks through their own reasoning ability. Existing benchmarks offer limited insight into this question as they often ignore intermediate reasoning steps, typically assess only final answers, and thus cannot know how models fail. To address this gap, we present CAPTCHA-X, the first unified evaluation framework for real-world CAPTCHA solving with reasoning steps, enabling automatic assessment of solving accuracy and reasoning quality. Using this framework, we show that reasoning substantially improves performance. Beyond evaluation, CAPTCHA-X exposes characteristic reasoning failure modes of vision-language models and reveals reasoning scaling laws that connect multiple dimensions of reasoning. Furthermore, evaluating pure vision generation models reveals an interesting phenomenon: generation models also fail at multi-step logical planning on images. These findings motivate our super-simple and reasoning-centric agent built solely on vision-language models, designed to identify and correct errors in intermediate reasoning steps before translating reasoning into executable actions.

Vision language models are predominantly evaluated on English-centric multimodal benchmarks, leaving their behavior in other languages and cultures insufficiently understood. We introduce VMMU, a Vietnamese Multitask Multimodal Understanding and Reasoning benchmark of 2,548 multiple-choice questions across seven domains: Mathematics, Physics, Chemistry, Biology, Geography, Driving Test, and IQ Test. Every question requires jointly reasoning over Vietnamese text and non-text visual evidence (charts, diagrams, tables, traffic scenes), and is distributed as a single rendered page image. We evaluate 10 open-source and 5 closed-source VLMs on VMMU. Test-time compute improves performance from 57% to 78% accuracy, but it remains inferior to an expert baseline (99% accuracy). Extensive error analyses show that closed-source VLMs already achieve strong Vietnamese OCR performance, yet still struggle on VMMU. This suggests that the primary bottleneck is multimodal grounding and reasoning rather than text recognition or Vietnamese language understanding. Code and data are available at: https://vmmu-bench.github.io.