Progress in question answering (QA) over tabular data has enabled reliable factual retrieval from relational tables. However, many real-world questions are probabilistic, requiring reasoning under uncertainty and latent conditional dependencies that are not directly stored in individual cells. We introduce LUCARIO, a large-scale benchmark for probabilistic QA over real-world tabular datasets, covering diverse domains, dependency patterns, and natural language variations. We propose Auto-BN, a fully automatic neuro-symbolic framework that induces a Bayesian Network from raw tables, translates natural-language questions into formal probabilistic queries, and performs exact inference. Experiments across multiple LLM backbones show that Auto-BN consistently outperforms NL2SQL, retrieval-based, and premise-based baselines, while maintaining a near-zero error rate. LUCARIO fills a relevant gap and provides a practical and scalable testbed for advancing probabilistic reasoning over structured data.

To protect patient privacy, medical models are frequently trained on synthetic datasets. Focusing on 13 clinical attributes commonly found in breast cancer records, we propose a synthetic data generation framework in which each generated sample is a data–report pair: the structured data capture the 13 attributes of interest, while the accompanying (French) text simulates a corresponding breast cancer pathology report. Data privacy and coverage are obtained by generating the structured data using a tabular data synthesiser. In this work, we focus on enhancing the diversity and quality of the generated texts through a combination of data-driven knowledge distillation, data filtering and data combination. Our findings suggest that the quality of the training data used for the student model outweighs its quantity; that combining data from multiple teachers helps improve performance; and that a compact model (Mistral 7B) can match the zero-shot Information Extraction performance of a significantly larger model (MedGemma 27B) when fine-tuned on high-quality synthetic data.

Large language model-based multi-agent systems (LLM-MAS) are designed through roles, debate protocols, and aggregation rules. These choices create implicit expectations of trust, skepticism, deference, or collaboration. We make inter-agent relations explicit as signed pairwise priors, rendered in natural language and added to system prompts while keeping the task protocol fixed. Across commons governance and multi-agent debate, these priors change how readily agents coordinate or agree, a pattern we call convergence pressure. More positive relations generally improve sustainability within the GovSim relational-prior sweep and increase consensus on subjective questions. These improvements over negative relations differ from gains over no-prior prompting. On objective QA, fully positive priors usually yield lower final-answer accuracy than the no-prior baseline, and some conditions produce more frequent but less accurate consensus. Effects depend on model backbone, relation type, and topology; explicitly neutral relations also produce different outcomes from omitting relational framing. Relational priors are therefore task-specific interventions and diagnostic probes of sensitivity to social framing. Evaluate them against a no-prior baseline using the task’s primary metric, report accuracy and consensus correctness alongside agreement on objective tasks, and keep no-prior prompting as the default for accuracy-centric tasks unless validation supports a relational prior.

Tracking semantic change in low-resource languages across extensive historical timelines presents significant challenges due to data scarcity and the limitations of static embedding alignments. This study investigates the diachronic evolution of the Sinhala language from the 13th to the 20th century using a multi-stage computational framework. We first align century-specific Word2Vec and FastText embeddings using Similarity Matrix Based Alignment (SMA) and Orthogonal Procrustes (OP) techniques, finding that OP alignment provides more stable neighbourhood tracking for identifying temporal similarity dips. To move beyond aggregate measures, we introduce a Bidirectional Semantic Impact Pruning approach using contextualised embeddings from a fine-tuned Llama-3.1-8B. By applying Leave-One-Out (LOO) diagnostics, we attempt to isolate influential sentences to distinguish between systemic semantic shifts and transient polysemic expansion. Our results show that semantic drift in the fine-tuned Llama-3.1-8B is not evenly distributed across all usages. Instead, a significant part of the change is driven by a smaller set of high-impact contextual instances, rather than gradual and uniform change across all occurrences. This work provides a preliminary framework for low-resource Sinhala diachronic analysis, highlighting the trade-offs between model sensitivity and data availability.

Visual Question Answering (VQA) over charts and tables requires joint reasoning over language, visual structure, and numerical information, yet no standard benchmark currently exists for Vietnamese. We introduce ChartTableViVQA, a Vietnamese VQA dataset with 13,003 question–answer–instruction triplets over 868 chart and table images. Questions cover multiple reasoning types, ranging from simple value lookup to multi-step aggregation, comparison, and trend analysis, and are accompanied by concise instructions that explicitly guide the reasoning process. The dataset is built via a semi-automatic human-in-the-loop pipeline, combining large language model generation with manual verification by Vietnamese annotators. We evaluate several vision-language models in zero-shot and instruction-based settings. Results show strong performance on simple questions but clear weaknesses in complex reasoning, particularly on charts, while instruction-based prompting consistently improves accuracy.

Automatic Speech Recognition (ASR) can ease clinical documentation in resource-constrained regions, but deployment is hindered by a "Reality Gap" between laboratory performance and noisy, real-world clinical telephony, compounded by strict data residency and compute constraints. We study this gap using Gram Vaani, a telephonic Hindi corpus spanning rural healthcare and agricultural helplines, as the closest publicly available proxy for clinical telephony speech, and show that a robust multilingual model (IndicWav2Vec) degrades from 11.60% WER on clean read Hindi to 41.72% WER on this data. We evaluate a progression of adaptation regimes, from full fine-tuning and offline Low-Rank Adaptation (LoRA) upper bounds to an on-device, stream-based continual adaptation framework in which raw audio never leaves the local device, and characterize the trade-offs between data-driven and parameter-driven stabilization strategies. Our evaluation covers both lexical accuracy (WER and CER) and semantic fidelity (BERTScore) on the target domain, alongside the retention of general-domain knowledge. Multi-domain Experience Replay (ER) yields the primary gains, improving target WER by 18.2% relative and reducing catastrophic forgetting by 54% compared to naive adaptation, with BERTScore reflecting consistent gains in semantic fidelity. Combining replay with Elastic Weight Consolidation based on a stabilized importance estimate (Absolute Fisher) yields the strongest retention at a small cost in plasticity. Finally, a language model spot check empirically verifies that the core mismatch lies at the acoustic level and cannot be resolved by language models alone.

Image Captioning lies at the center of synthetic data generation in computer vision. The quality of the caption defines the behavior of the resulting model trained on image-caption pairs generated by Image Captioning. Measuring the specificity of image captioning plays an important role in supplying high-quality captions for the downstream tasks. To address the challenge of automatically assessing the quality of the caption, we propose SPECI, an automatic metric that decomposes dense captions into atomic facts and evaluates specificity across visual criteria such as granularity, spatial detail, color, and linguistic focus using WordNet-based hyponym levels. Empirical results on the benchmark datasets demonstrate significant correlation with human judgments, and its application in image captioning and image reconstruction tasks highlights the practical utility of SPECI. The source code of SPECI is available at https://github.com/sonlam1102/SPECI.

Large language models (LLMs) exhibit both knowledge of language, rendered by, for example, grammatical judgment ability, and abilities to perform tasks through language, such as math-word problem solving. Motivated by neurolinguistic studies of their neural dissociation in human brains, in this study, we investigate the overlap of their circuits in large language models (LLMs). Specifically, we examine the overlap and causal effects between the circuits extracted for linguistic and arithmetic tasks, using controlled materials and circuit discovery methods from the mechanistic interpretability field. Our experiments with five LLMs show that circuit similarity is mostly explained by task domains (linguistic vs. arithmetic), rather than by their input and output vocabularies or by the syntactic similarities between natural-language sentences and arithmetic expressions. This finding suggests that LLMs may process linguistic and arithmetic information using largely separate neural circuits, highlighting an ability-based modular organization of circuits within these models.

Activation steering offers a lightweight mechanism for controlling model behavior, but existing methods depend on sentence-level contrastive data that is unavailable for most of the world’s languages. We show that multilingual sentiment lexicons are a viable substitute across three model families: aggregated multilingual lexicons outperform English-derived steering vectors for mid- and low-resource languages, while monolingual lexicons remain stronger for languages a model has specialized in during pre-training. The steering efficacy is mechanistically dictated by both representational density and pre-training priors. Large-magnitude English vectors overwhelm the compact representations of low-resource languages, causing "linguistic collapse." This fragility is mitigated by the norm-dampening of multilingual aggregation. Architectural biases, including bilingual specialization penalty and latent numerical rigidity, also heavily constrain cross-lingual steering responsiveness. Ultimately, our work establishes lexicons as a practical steering signal and offers a comprehensive mechanistic account of cross-lingual steering dynamics.

Understanding the impact of prompt variations on large language model (LLM) outputs is critical. Despite extensive research, two challenges persist: conflated terminology stemming from loosely defined concepts, and inefficient evaluations reliant on output generation. To address these, we first formalize prompt categorization to clearly define prompt robustness and controllability. We then introduce the prompt vector (PV), a compact internal representation that reformulates these properties at the representation level. Building on PV, we propose PRC, an efficient and unified framework to systematically evaluate LLMs under few-shot prompt variations without requiring costly autoregressive decoding. Experiments across diverse instruction-tuned LLMs demonstrate that PRC tightly correlates with traditional output-based metrics while substantially reducing computational overhead. Overall, PRC offers a principled and highly efficient tool for diagnosing LLM behavior.

Human-AI collaboration on complex planning goals is bottlenecked by how LLM interfaces handle context: users must manually curate and re-surface relevant information across long and unstructured chat histories. Despite advances in long-context prompting and memory-augmented retrieval, this burden remains unresolved: users still have to identify and supply the right context at each decision point, regardless of how much the model can store or surface. We propose JumpStarter, a system that enables LLMs to collaborate with humans on complex goals by dynamically decomposing tasks to help users manage context. We specifically introduce task-structured context curation, a framework that breaks down a user’s goal into a hierarchy of actionable subtasks and scopes context to localized decision points, enabling finer-grained personalization and reuse. The framework is realized through three core mechanisms: context elicitation, selection, and reuse. In a within-subjects user study, plans produced with JumpStarter were rated substantially higher in quality than those produced with ChatGPT. A complementary automatic simulation study shows that JumpStarter consistently outperforms ChatGPT baselines, planning and memory agents, and workflow ablations. These findings show that effective human-AI planning depends not on the volume of context provided, but on attaching the right context to the right subtask at the right time.

Large language models are widely used for personal advice (relationship conflicts, financial decisions, mental-health support, etc.), but they often align their response with the user’s position rather than offering independent judgment, even when users need pushback rather than validation. Mitigating this, whether through training objectives, system-level guardrails, or evaluation design, requires knowing what triggers it. But what in the user’s message triggers this behavior is unknown. Natural advice-seeking messages vary simultaneously in semantic content (what is disclosed), linguistic style (how it is written), and narrative framing (whose side is told), but existing benchmarks cannot attribute model behavior to any single cue. Without this attribution, mitigations cannot be targeted, for example, suppressing warmth may harm users with genuine distress but might not fix the model’s tendency to endorse whichever side of a conflict speaks. In this work, we isolate which of these cues drive which aspect of sycophantic behavior across 787 advice-seeking scenarios and observe three patterns emerge: emotional warmth is content-driven; stance endorsement is framing-driven: endorsement drops by ∼50% when the same conflict is retold from the opposing perspective; and recommendation directness is style-driven. We also find that hedging reduces rather than increases model endorsement; models do not simply validate vulnerable-sounding users.

The exponential growth of scientific literature has made personalized, factually grounded paper discovery increasingly difficult. Existing systems suffer from limited personalization, cold-start fragility, or hallucinated explanations. We introduce HybridScholar, an integrated architecture combining a CrossAttention-gated fusion of collaborative filtering and RAG learning candidate-specific gate weights per candidate with a Chain-of-Verification (CoVe) pipeline for factually grounded explanations. Evaluated on the Scholar Inbox dataset, HybridScholar substantially outperforms (NDCG@10=0.2991) strong baselines including LightGCN and Scholar-Inbox. Cold-start analysis shows the RAG fallback prevents hard failure for sparse users (NDCG@10=0.283), with cold NDCG rising monotonically as interactions grow. The CoVe pipeline reduces hallucination rate by 23.8%, confirmed by a human evaluation study showing CoVe-revised explanations are preferred by experts on factual accuracy (76.7% vs. 36.0%) and by end-users on helpfulness (78.8%, p<0.0001). Overall, HybridScholar improves ranking quality, cold-start robustness, and factual reliability for large-scale scholarly research discovery, demonstrated on 15,881 users across 239,618 papers.

LLMs are known to introduce stylistic changes into generated text, yet how these stylistic shifts affect model decisions on scientific tasks remains underexplored. In this paper, we focus on multimodal claim verification, where the goal is to determine whether a textual claim is grounded in a given piece of evidence. We apply two rewriting strategies: natural rewriting, which simulates how researchers routinely use LLMs to polish academic text, and controlled injection, which inserts a single LLM-associated word to isolate the effect of vocabulary choice. We evaluate 11 open-weight models spanning five VLM families and ranging from 2B to 38B parameters. We find that models are robust to these modifications: most show no significant drop in accuracy, and compared to prior work on review-score manipulation, verification appears far more stable. However, consistent probability shifts do occur. Hedging-oriented conditions produce significant shifts across nearly all models, while boosting conditions show a weaker effect and general polishing conditions (e.g., grammar correction, fluency improvement) have little effect.

Argument summarization seeks to generate concise representations of argumentative content while preserving argumentative meaning, yet the field remains conceptually fragmented. In this survey, we provide a unified task definition of argument summarization that encompasses both argumentative text summarization and key point analysis, and introduce a taxonomy along input structure, output representation, and the role of argumentative structure. Using this framework, we analyze the evolution of the field and systematically review existing datasets, methods, and evaluation practices, revealing misalignment that hinders comparability and progress. We further situate argument summarization within the broader context of generic text summarization, identifying practical limitations and intrinsic challenges. Finally, we outline promising future research directions.

Dense retrievers can blur the distinctions between short, jargon-heavy records that share most of their vocabulary. We study Lex-TM, a training-free layer that interpolates dense cosine similarity with the Jensen–Shannon similarity of topic distributions from an LDA model whose topic–word prior is scaled by a Herfindahl–Hirschman (HHI) exclusivity score. On NTCIR-13 MedWeb, cast as retrieving tweets that share a query’s exact symptom-label set, Lex-TM reaches an English MRR of 0.8151 against 0.7081 for BM25+dense reciprocal rank fusion, a significant paired difference; in Japanese and Chinese the differences from fusion are not significant. With Qwen3-Embedding-0.6B, routing improves on dense retrieval significantly in English and Chinese. Three findings bound these gains. SPLADE++ and BGE-M3’s multi-vector mode match or exceed Lex-TM. In English and Chinese the gain depends on the label strings in the indexed tweets: without them, routing no longer improves on dense retrieval. The HHI prior brings no benefit: on short texts it becomes a rarity weight, and a symmetric prior performs as well across five Gibbs seeds. Outside MedWeb, routing matches dense retrieval on Chinese medical QA and lowers nDCG@10 on ChatDoctor, FiQA and long biomedical abstracts. Code, seeds and a changelog of fixed bugs are released.

We present a new dataset to support AI-assisted and human studies of the peer reviewing process, based on the ARR Open Review Platform. Unlike previous similar data collections, ours, ARR-Dv2, not only contains the reviews but also author-reviewer discussions (aka rebuttal phase) and meta-reviews. We present statistical analyses of the data with respect to the overall score and the correlations among the overall score, confidence, and soundness scores. Additionally, we extend the task of Review Score Prediction also to include rebuttal data and analyze its effect on score prediction. Finally, we introduce a new task, Meta Review Score Prediction, based on a set of up to three reviews rather than a single review. Our initial results in a zero-shot setting indicate that the rebuttal data adds valuable information to the score prediction and enables reliable predictions.

Contrastive decoding methods such as DoLa improve the factuality of Large Language Models (LLMs) by contrasting the output distributions of mature and premature layers. However, DoLa’s dynamic layer selection relies solely on divergences in output vocabulary distributions. In this work, we propose three attention-guided strategies—Attention-JSD, Attention-Entropy-Max, and Attention-Entropy-Min—that leverage structural information carried by internal self-attention mechanisms as a signal for layer selection. Experimental results on TruthfulQA demonstrate that our strategies, particularly Attention-JSD and Attention-Entropy-Min, consistently outperform the original DoLa. We observe significant gains on multi-answer metrics (MC2 and MC3), suggesting that attention distributions can provide a more sensitive signal for resolving factual knowledge than output vocabulary distributions.

Long-term memory in Large Language Model (LLM) agents requires selective persistence: only a subset of interactions should be consolidated beyond the current context window. Existing memory systems rely on heuristic importance rules or similarity-based novelty, which remain external to the model’s internal computation. We propose a geometric principle for memory formation: episodic memory can emerge from compression failure in latent representation space. We approximate the manifold of routine LLM activations using Sparse Autoencoders (SAEs) and define representational surprise as reconstruction error relative to this learned manifold. Deviations from routine structure yield elevated residuals, providing an unsupervised, model-internal signal for memory writing. We demonstrate this principle through ReSuME, a surprise-gated memory mechanism that commits turns to memory only when normalized reconstruction error exceeds a calibrated threshold. In long-horizon multi-turn dialogue settings, representational surprise separates routine, critical, and out-of-distribution states, and achieves a superior performance–memory trade-off against established memory-agent systems (MemGPT, EM-LLM) and unsupervised anomaly detectors, while operating without labels and at substantially lower inference cost than LLM-judge controllers. Covariance-aware normalization further enables robust cross-domain calibration. These results suggest that episodic memory gating in neural agents can be grounded in intrinsic latent geometry rather than externally engineered rules.

Large Language Models (LLMs) are now widely used for tasks such as summarization, question answering, and speech-to-text translation, and increasingly for code generation, with tools such as Cursor and Windsurf analyzing massive code repositories and recommending changes. Big tech companies have acknowledged growing reliance on LLM-generated code. Although this improves developer productivity, it also raises the risk of suboptimal and insecure code. We introduce a retrieval framework for code generation that connects geometry-aware prototype sampling with syntax-grounded interpretability analysis. By conditioning AST-based explanations on retrieved demonstrations, our method evaluates retrieval quality through both functional correctness and interpretability alignment. In our experiments, high-quality in-context learning (ICL) demonstrations make outputs easier to interpret and improve P@10, whereas poorly chosen demonstrations reduce P@10 relative to the base model. Overall, our approach highlights the importance of efficient sampling strategies for ICL, which can affect model performance on any given task.

Large Language Models (LLMs) have made substantial progress in multilingual question answering, yet their ability to understand low-resource sociolinguistic contexts remains underexplored. We study North Korea as a unique testbed where decades of division have produced distinct differences from South Korea in vocabulary, orthography, norms, and ideological discourse. We introduce LENS (Language and sociocultural Evaluation for North Korean Society), a human-validated benchmark of 1,765 questions evaluating LLMs across three dimensions: linguistic robustness, sociocultural perspective alignment, and domain-specific knowledge. Evaluating diverse recent LLMs, we find a modest performance drop on North Korean language inputs, a larger gap in sociocultural perspective alignment, and severe limitations on domain-specific knowledge. Qualitative analysis reveals that models often conflate formal institutions with lived realities, project liberal-democratic assumptions, or rely on coarse stereotypes. Our results demonstrate that linguistic similarity alone does not guarantee sociocultural or domain-specific understanding, highlighting the critical need for grounded evaluation resources for underrepresented cultural and political contexts.

Large language models (LLMs) are increasingly deployed across high-impact domains, from clinical decision support and legal analysis to hiring and education, making fairness and bias evaluation before deployment critical. However, existing evaluations lack grounding in real-world scenarios and do not account for differences in harm severity, e.g., a biased decision in surgery should not be weighed the same as a stylistic bias in text summarization. To address this gap, we introduce HALF (Harm-Aware LLM Fairness), a deployment-aligned framework that assesses model bias in realistic applications and weighs the outcomes by harm severity. HALF organizes evaluation datasets into harm tiers (Severe, Moderate, Mild) based on the specific task and bias type they measure. Harm severity is assigned according to deployment-relevant criteria grounded in regulatory and risk assessment frameworks. Our evaluation results across eight LLMs show that (1) LLMs are not consistently fair across datasets, (2) model size or performance do not guarantee fairness, and (3) reasoning models perform better in medical decision support but worse in education. We conclude that HALF exposes a clear gap between previous benchmarking success and model fairness. All data and code are available at https://github.com/mbzuai-nlp/HALF

Understanding the multilingual mechanisms of large language models (LLMs) provides insight into how they process different languages, yet this remains challenging. Existing studies often focus on individual neurons, but their polysemantic nature makes it difficult to isolate language-specific units from cross-lingual representations. To address this, we explore sparse autoencoders (SAEs) for their ability to learn monosemantic features that represent concrete and abstract concepts across languages in LLMs. While some of these features are language-independent, the presence of language-specific features remains underexplored. In this work, we introduce SAE-LAPE, a method based on feature activation probability, to identify language-specific features within the feed-forward network. We find that many such features predominantly appear in the middle to late layers of the model and are interpretable. These features influence the model’s multilingual performance and language output, and can be used for language identification with performance comparable to fastText, along with more interpretability. Our code and complete figures are available [here](https://github.com/LyzanderAndrylie/language-specific-features).

Current LLM-based question answering systems provide cited evidence for their claims. However, determining whether each claim within the generated responses is fully supported by its cited evidence is still an ongoing research problem. The task of evaluating attribution quality, or faithfulness to the source text, goes beyond the traditional idea of lexical matching used for relevance in early QA tasks, and the use of lexical models based on semantic distance or word-overlap has declined in favor of NLI-based models and large fine-tuned models. The larger fine-tuned models tend to perform best on current special-purpose benchmarks such as AttributionBench despite struggling with OOD data. Still, exactly why human judgments of faithfulness should align better with LLM evaluators than those based on lexical entailment has never been fully explained. We address the problem with current entailment models by introducing bidirectionality and codifying that idea into a new evaluation modeling approach. We show that this approach is both lightweight and extensible yet competitive with SOTA LLM-based methods, avoiding the pitfalls of both standard NLI and word-overlap approaches. We further show that our model architecture is motivated by a deeper understanding of the gap between lexical distance and faithful claim attribution, exposing the conceptual mechanism behind attribution quality evaluation.

Large language models are increasingly used to annotate texts, but their outputs reflect some human perspectives better than others. Existing methods for correcting LLM annotation error assume a single ground truth. However, this assumption fails in subjective tasks where disagreement across demographic groups is meaningful. Here we introduce Perspective-Driven Inference, a method that treats the distribution of annotations across groups as the quantity of interest, and estimates it using a small human annotation budget. We contribute an adaptive sampling strategy that concentrates human annotation effort on groups where LLM proxies are least accurate. We evaluate on politeness and offensiveness rating tasks, showing targeted improvements for harder-to-model demographic groups relative to uniform sampling baselines, while maintaining coverage.

Large Language Model (LLM)-based UI agents show great promise for UI automation but of- ten hallucinate in long-horizon tasks due to their lack of understanding of the global UI transition structure. To address this, we in- troduce AGENT+P, a novel framework that leverages symbolic planning to guide LLM- based UI agents. Specifically, we model an app’s UI transition structure as a UI Transi- tion Graph (UTG), which allows us to refor- mulate the UI automation task as a pathfinding problem on the UTG. This further enables an off-the-shelf symbolic planner to generate a provably correct and optimal high-level plan, preventing the agent from redundant explo- ration and guiding the agent to achieve the au- tomation goals. AGENT+P is designed as a plug-and-play framework to enhance existing UI agents. Evaluation on the AndroidWorld benchmark demonstrates that AGENT+P im- proves the success rates of state-of-the-art UI agents by up to 14.32% and reduces the ac- tion steps by 37.70%. Our code is available at: https://github.com/KiteFlyKid/agentp.

Large language models are now deployed as annotators across NLP, generating training labels and evaluation scores, yet the field has no established standard for verifying whether LLM-generated labels are reliable, internally consistent, or comparable across studies. This methodological gap does not begin with LLMs: a systematic corpus study of 665 ACL Anthology papers from 1999 to 2026 reveals that even classical human annotation in NLP suffers from systematic measurement and reporting failures that have left a growing body of work scientifically incomparable. Papers were identified through keyword search and organised via topic modelling into a thematic taxonomy spanning eight annotation domains from named entity recognition to disagreement modelling. Full-text analysis of 549 papers yields three principal findings. First, Cohen’s kappa dominates across all task types (259 papers, 38.9%), regardless of whether its assumptions are satisfied; ICC, appropriate for continuous-scale tasks such as MT evaluation, appears in only 16 full-text papers (2.4%). Second, 226 of the 665 papers (34.0%) name no inter-annotator agreement (IAA) coefficient anywhere in the full text, which is a methodology-reporting concern; a further 50.1% of papers name a coefficient in the body but not in the title or abstract, so 84.1% of papers omit the metric from the abstract layer. We frame the body-only gap as a discoverability concern (the methodology is reported but is not visible at the abstract layer readers and reviewers encounter first), distinct from the full-text omission. Third, annotator disagreement modelling and LLM-as-annotator constitute the fastest-growing area of the corpus (124 papers, 18.6%): 26 of these appeared between 1999 and 2022 and 98 between 2023 and 2026, the only dimension whose volume rises in that window. Fewer than 5% of papers report confidence intervals. We provide a large-scale empirical analysis of metric reporting practice within the ACL Anthology and a seven-point agenda for standardised reporting, complementing prior normative guidance with field-scale descriptive evidence.

This paper presents a case study on the development of Auto-AdvER, a specialised span-labelling schema, along with its corresponding dataset for the textual car advertisement domain. Developed with industry needs in mind, Auto-AdvER is designed to enhance text mining analytics in this domain and contributes a linguistically unique sequence-labelling dataset. We introduce a schema consisting of three labels: ’Condition’, ’Historic’, and ’Sales Options’. We outline the guiding principles for annotation, describe the methodology for schema development, and present the results of an annotation study demonstrating an inter-annotator agreement of 78%. Furthermore, we compare the performance of various models, including transformer encoder models (BERT, DeBERTaV3) and transformer decoder-based open- and closed-source large language models: Llama, Qwen, GPT, and Gemini, as well as conditional random fields. Our results show that a fine-tuned DeBERTaV3 encoder achieves the highest F1-score, followed closely by conditional random fields, while the large language models reach the highest precision but lower recall. We present this work as a stepping stone towards more fine-grained analysis and discuss Auto-AdvER’s potential impact on advertisement analytics, pricing, and customer insights. Potential applications include the analysis of market dynamics and data-driven predictive maintenance.

Vision-Language Models (VLMs) enable video annotation at scale, but costs accumulate quickly: processing a typical 60-second short-form video at one frame per second requires millions of tokens. To reduce costs, researchers rely on heuristics such as sampling a subset of frames, compressing videos into image grids, or using only a single modality. However, it remains unclear which heuristics save cost, and whether they preserve the downstream conclusions these annotations enable. To address this gap, we conduct a systematic evaluation of these heuristics using short-form videos, on two computational social science (CSS) tasks: sentiment and topic classification. We evaluate each configuration along three axes the literature typically treats separately: classification accuracy, validity of downstream inference, and per-video token cost. First, we find that accuracy and validity diverge: the highest-accuracy configuration can produce wrong conclusions. Second, modality value is not guaranteed: text alone can yield strong performance, indicating that adding modalities can add cost without adding signal. Finally, we find that cost can be decoupled from video length when annotating short-form videos: a single 2x8 image grid built via simple shot-transition detection approaches full-video understanding (kappa within .05), at ~15% of the token cost. Based on these findings, we derive guidelines that can enable cost-aware VLM annotation in CSS.

Recent advancements in large language models have introduced new capabilities for reasoning over structured data, particularly through program-aided tools that can analyze tables. However, many existing methods address single-table scenarios or assume that the relevant tables are already provided. In practice, users often issue complex data exploration queries over entire databases, where relevant information may be distributed across multiple relations. In this work, we introduce DBRAG, a retrieval-augmented generation framework tailored for multi-table question answering. DBRAG first retrieves candidate tables using an offline table index, enriches their summaries with query-relevant rows, and uses an LLM to rerank the candidates. A program-aided reasoner then selects the required tables and executes operations over their full contents, keeping the initial prompt context compact. Experiments on the Spider, GeoQuery, and ATIS datasets used in this study demonstrate improvements in table retrieval and multi-table question answering.

Multi-Agent (MA) systems are effective at solving complex tasks that demand planning, tool use, and the synthesis of evidence from multiple sources. Existing systems typically adopt Hierarchical Manager-Worker (HMW) or Router-based Message Passing (RMP) structures as their communication protocol. However, these designs restrict agent autonomy: Worker agents cannot directly consult specific “peers”, and misrouted messages can propagate errors. Inspired by bus architectures in computer systems, we propose BusMA, a communication framework that allows any agent to address other agents through a shared channel, i.e., the Bus. It consists of agent registration, message routing, and shared memory management components. Worker agents, each equipped with tools, have their own local memory and can reason, act (tool usage), and communicate by posting shared messages with specific intents. We introduce four intents: discussion, challenge, guidance, and request for explanation, which support fine-grained communication among agents. A Chair agent monitors the shared memory to coordinate interactions and facilitate convergence among Workers. To evaluate the effectiveness of BusMA, we conduct extensive experiments with two frontier LLMs across 13 tasks spanning visual reasoning, mathematical reasoning, and knowledge retrieval. The results demonstrate that BusMA consistently outperforms state-of-the-art HMW and RMP methods.

In visually-rich documents, information is encoded not only in individual page objects such as tables, headers, and text blocks, but also in the structural relations among them, making document structure analysis fundamental to information retrieval and document understanding. However, accurately inferring such relations remains challenging in multi-page documents with long-range dependencies and heterogeneous layouts. To address this, we propose a tree-guided and self-regressive framework, termed DOcument Structure Analyzer (DOSA), for inferring relations among page objects and reconstructing document-level semantic trees. DOSA processes documents chunk-by-chunk, fusing visual, textual, and layout features for each page object and predicting hierarchical and ordering relations. The predicted relations are used to incrementally construct a semantic tree, which is then leveraged as structural context to guide inference on subsequent chunks. Experimental results on five benchmarks demonstrate the effectiveness of DOSA, with improvements of up to 4 F1 points and 19 TEDS points on DocHieNet, the most challenging multi-page hierarchy benchmark.

Discrete diffusion language models can generate text efficiently by updating multiple masked positions in parallel, but this parallelism introduces a quality-latency trade-off. Aggressive decoding may commit mutually dependent tokens too early, while conservative decoding requires many denoising steps. Existing methods address this tension by deciding which tokens are safe to reveal using confidence or dependency criteria. However, avoiding unsafe commits does not necessarily make the remaining masked sequence easy to decode, since uncertain tokens may depend on masked tokens, creating a bottleneck for denoising steps. We propose AXON, a training-free module that can be added on top of existing parallel decoding strategies for diffusion language models. Rather than replacing the base decoder, AXON monitors the remaining uncertain masked tokens and intervenes only when their current state suggests that additional context is needed. It then shifts the criterion from which tokens are safest to reveal to which confident reveals would best support later denoising. AXON selects anchors, confident masked tokens that uncertain positions attend to, using attention, uncertainty, and confidence signals. Experiments on reasoning and code-generation benchmarks across multiple diffusion language models show that AXON improves the quality-latency trade-off of existing parallel decoders, often reducing the number of function evaluations while maintaining or improving accuracy.

Black-box distillation trains student models using only teacher-generated outputs, making it practical when teacher models are accessible only through APIs. However, existing methods either rely mainly on positive imitation or require costly full-response rollouts to obtain corrective signals. We show that such corrective signals can be obtained locally from teacher trajectories alone. At each position, the teacher token provides a positive anchor, while a high-probability alternative sampled from the student provides an informative negative. Based on this insight, we propose Token-level On-policy contrastive Learning for blackbox Distillation (TOLD), which samples a single negative token from the student’s distribution and optimizes a Bradley-Terry objective anchored by negative log-likelihood. This enables the student to reinforce teacher tokens while suppressing its own likely mistakes, without teacher logits, reward models, or full-response rollouts. Experiments on reasoning and instruction-following tasks show that TOLD outperforms or matches strong black-box distillation baselines and achieves up to 1.88 times training speedup over rollout-based on-policy baselines.

Autonomous agents based on large language models (LLMs) are becoming an increasingly prevalent paradigm for tackling complex and real-world tasks. Despite the remarkable zero-shot capabilities of modern LLMs, specialized agent training is often essential to obtain reliable and improved performance for specific target tasks. In this survey, we provide a process-centric review of the training of LLM-based agents, with a primary focus on methods that update the parameters of the main agent policy. Rather than organizing prior work only by application domain, agent capability, or learning algorithm, we examine the end-to-end pipeline through which LLM-based agents are trained: environment setup, data and trajectory construction, learning-signal design, and optimization objectives and schemes. This perspective highlights how design choices at different stages interact with one another. Finally, we conclude with a discussion on potential future directions.

Despite advances in safety alignment, prompt-rewriting attacks such as persona modulation, fictional framing and persuasion-based reformulation, can bypass safety filters even on frontier models. Existing defenses either rely on non-scalable human curation or white-box optimisation that overfits to specific model internals, leaving aligned models brittle against the very class of adaptive black-box adversaries they will face in deployment. To address this gap, we introduce CHASE (Co-evolutionary Hardening through Adversarial Safety-Escalation), a closed-loop red-blue teaming framework in which a black-box attacker and a safety-aligned defender co-evolve. The attacker is trained via Group Relative Policy Optimization (GRPO) under a multiplicative reward that jointly enforces bypass effectiveness and intent fidelity, while the defender is hardened on the harvested adversarial rewrites through a two-stage GRPO + rejection-sampled SFT pipeline balanced with benign data. Evaluated on BeaverTails and JailbreakBench against five held-out attack families (PAIR, TAP, AutoDAN, PAP, Translation), CHASE cuts mean StrongREJECT score by 43.2% on Llama-3.1-8B-Instruct with 0% false refusal on benign (Alpaca) prompts, at a measured helpfulness cost of a 55.6% refusal rate on XSTest safe prompts and a 1.92-point MT-Bench reduction. A controlled ablation that varies only the training attack distribution shows that this generalisation comes from the template-free attack distribution rather than the pipeline, and the result replicates on Qwen2.5-7B-Instruct. Across the two similarly sized open models we study, these results suggest that template-free RL exploration recovers latent attack primitives that transfer across mechanistically distinct attack families.

Personalized text generation requires models not only to produce coherent text but also to align with a target user’s style, tone, and topical focus. Existing retrieval-augmented approaches such as LaMP and PGraphRAG enrich profiles with user and neighbor histories, but they stop at generation and often yield outputs that drift in tone, topic, or style. We present PerFine, a unified, training-free critique–refine framework that enhances personalization through iterative, profile-grounded feedback. In each iteration, an LLM generator produces a draft conditioned on the retrieved profile, and a critic LLM - also conditioned on the same profile - provides structured feedback on tone, vocabulary, sentence structure, and topicality. The generator then revises, while a novel knockout strategy retains the stronger draft across iterations. We further study additional inference-time strategies such as Best-of-N and Topic Extraction to balance quality and efficiency. Across Yelp, Goodreads, and Amazon datasets, PerFine consistently improves personalization over PGraphRAG, with G-Eval gains of +8.5–12%, steady improvements over 3–5 refinement iterations, and scalability with increasing critic size. These results highlight that post-hoc, profile-aware feedback offers a powerful paradigm for personalized LLM generation that is both training-free and model-agnostic.

Recent advances in Large Language Models (LLMs) have expanded their role in human interaction, yet information-seeking behavior, particularly across cultural contexts, remains underexplored in these systems. In this work, we study cross-cultural variation in information-seeking using question-asking as a mechanism. We introduce IQUEST (Information-seeking QUestion Evaluation across SocieTies), an evaluation framework that measures human-model alignment in information-seeking question behavior through linguistic (style), topic preference (content) analysis and grounding insights in social science constructs. Across open- and closed-source models, we find that LLMs flatten cross-cultural diversity, aligning more closely with question-asking patterns expressed in Western countries. We also explore fine-tuning strategies, which narrow the human-model alignment gap by 43% overall. Finally, we demonstrate that stronger information-seeking question behavior improves downstream cultural adaptability, highlighting its importance for inclusive NLP research.

Efficient context compression is critical for retrieval-augmented question answering in resource-constrained settings, where long retrieved contexts increase latency, memory use, and LLM reader cost. We propose a lightweight encoder-only framework for query-driven sentence pruning that preserves answer-critical evidence while aggressively reducing irrelevant context. Our method learns marginal contribution scores for sentences using counterfactual training signals and optimizes a contrastive ranking objective that separates critical evidence from noncritical context. Our approach scores all sentences from a single full-context encoding, enabling fast inference with low computational overhead. Experiments show that it maintains accuracy comparable to the strongest baseline while using 3.7× less peak memory and achieving nearly 3× lower compression latency, demonstrating an

Abstention is the intentional refusal of models to provide an answer. Adverse drug reactions (ADRs), which range from mild to life-threatening, are often underreported by patients in hospitals and to doctors because patients assume these are unavoidable. With the growing adoption of LLMs and vision language models (VLMs) in healthcare, patients are increasingly reaching out to these models for solutions. In such high-risk scenarios, incorrect or overconfident responses can cause unnecessary anxiety, lead patients to discontinue essential medications, or encourage inappropriate self-medication. In this work, we propose the MAD-MED, a Model Abstention approach for Drug Adverse Reaction for MEDical-VLMs. It is a training strategy to encourage models to abstain when answering would contradict the available input, exceed the model’s knowledge, or violate model ethics. In addition, we release an augmented Adverse Drug Reaction ABStention (ADR-ABS) dataset comprising image-answerable, text-answerable, and unanswerable questions to support systematic evaluation of abstention behavior. Quantitative and human evaluations confirm that our proposed MAD-MED approach improved the abstention capabilities of models considerably while keeping the generative capabilities intact in cases where the model is expected to answer.

Chinese text correction has traditionally focused on spelling and grammar, while factual error correction is usually treated separately. In paragraph-level Chinese professional writing, linguistic (word/grammar/punctuation) and factual errors can co-occur and interact. Moreover, editorial review leaves few observable errors in published professional texts, making controlled benchmark construction important. This paper introduces **CLFEC (Chinese Linguistic & Factual Error Correction)**, a new task for joint linguistic and factual correction. We construct a mixed, multi-domain Chinese professional writing dataset spanning current affairs, finance, law, and medicine. We then conduct a systematic study of LLM-based correction paradigms, from prompting to retrieval-augmented generation (RAG) and agentic workflows. The analysis reveals practical challenges, including limited transfer of a specialized corrector, the need for evidence grounding for factual repair, the difficulty of mixed-error paragraphs, and over-correction on clean inputs. Results further show that handling linguistic and factual errors within the same context outperforms decoupled pipelines, and that agentic workflows can be effective with suitable backbone models. Overall, CLFEC provides a new benchmark for Chinese text correction research and practical guidance for proofreading systems. The dataset is publicly available at [https://github.com/jiu2021/CLFEC-Dataset](https://github.com/jiu2021/CLFEC-Dataset).

Legal RAG systems often retrieve passages by semantic relevance, lexical score, or citation match. In versioned legal corpora, however, a passage can be topically relevant and citation-matched while being legally invalid for the operative date. We study law-in-force retrieval: retrieving evidence that matches the authority, is valid at the operative date, and supports the answer. We build LawInForce-eCFR from 805 human-validated eCFR/CFR amendment pairs and 3,220 controlled answer-flipping yes/no questions. We also present AuthorityFirst-RAG, a transparent diagnostic retriever that decomposes retrieval into authority identification, temporal scoping, and rule-support ranking. AuthorityFirst-RAG improves Recall@1 from 2.80% for BM25-date-filter to 30.90%, but generation remains difficult even with GoldOracle evidence. These results show a dual bottleneck in versioned legal RAG: retrieving the law in force and applying numeric legal rules.

Large Language Models (LLMs) are fluent but prone to hallucinations, producing answers that appear plausible yet are unsupported by available evidence. This failure is especially problematic in high-stakes domains where decisions must be justified by verifiable information. We introduce EvidenceRL, a reinforcement learning framework that enforces evidence adherence during training. EvidenceRL scores candidate responses for grounding (entailment with retrieved evidence and context) and correctness (agreement with reference answers) and optimizes the generator using Group Relative Policy Optimization (GRPO). We evaluate across two high-stakes domains, cardiac diagnosis and legal reasoning, where EvidenceRL consistently improves evidence grounding and faithfulness without sacrificing task accuracy. On cardiac diagnosis, F1@3 increases from 37.0 to 54.5 on Llama-3.2-3B while grounding (G_max@3) rises from 47.6 to 78.2; hallucinations drop nearly 5x and evidence-supported diagnoses increase from 31.8% to 61.6%. On legal reasoning, EvidenceRL raises Faithfulness from 32.8% to 67.6% on Llama-3.1-8B, demonstrating consistent behavioral change across domains.

While individual components of agentic architectures have been studied in isolation, we lack empirical evidence on how design dimensions interact in realistic multi-agent systems. This study aims to address these gaps by providing a comprehensive enterprise-specific benchmark. We introduce AgentArch: a benchmark that evaluates 18 agentic configurations across state-of-the-art large language models. The study spans four dimensions: orchestration strategy, agent style (ReAct versus function calling), memory architecture, and thinking tool integration. Results reveal strong model-specific architectural preferences that challenge one-size-fits-all assumptions in agentic AI. They also expose notable performance gaps on enterprise tasks: the best models reach only 35.3% success on the complex task and 70.8% on the simpler task. We aim to inform architecting choices for enterprise agents by grounding model selection and component design in systematic evidence.

Large Language Models utilizing reasoning techniques improve task performance but incur significant latency and token costs due to verbose generation. Existing automatic prompt optimization(APO) frameworks target task accuracy exclusively at the expense of generating long reasoning traces. We propose Cost-Regularized Optimization of Prompts (CROP), an APO method that introduces regularization on response length by generating textual feedback in addition to standard accuracy feedback. This forces the optimization process to produce prompts that elicit concise responses containing only critical information and reasoning. We evaluate our approach on complex reasoning datasets, specifically GSM8K, LogiQA and BIG-Bench Hard. We achieved an average 68% reduction in token consumption across diverse reasoning benchmarks (and up to 80.6% on algorithmic tasks) while maintaining competitive accuracy, seeing only a nominal decline in performance. This presents a pragmatic solution for deploying token-efficient and cost-effective agentic AI systems in production pipelines.

LLM-as-a-Judge frameworks offer scalable alternatives to human evaluation but inherit the limitations of text-only reasoning, particularly in tasks requiring precise computations. We introduce J-ToRL, a framework that trains LLM judges to leverage computational tools via code generation through end-to-end reinforcement learning (RL). Unlike prior approaches relying on inference-time integration, J-ToRL internalizes tool-use strategies as part of its policy, enabling exploration of diverse reasoning pathways. We identify a key challenge in applying RL with verifiable rewards to judgment: the coarse reward signals inherent in multiple-choice verdicts can incentivize shortcut learning. J-ToRL mitigates this via an auxiliary verifiable reward mechanism based on reference-answers, that enforces stricter criteria beyond simple verdict accuracy alone. With a sophisticated pairwise comparison dataset across five domains, J-ToRL-trained LLMs improve the base models, outperform prior judges, and close the gap to frontier LLMs. Further, J-ToRL-trained models exhibit beneficial cognitive behaviors such as self-error-recovery and self-refinement of tool use strategies.

Personalized multi-document summarization (MDS) is essential for meeting individual user preferences of writing style and content focus for summaries. In this paper, we propose that for effective personalization, it is important to identify fine-grained differences between users’ preferences by comparing preferences of different users. Motivated by this, we propose ComPSum, a personalized MDS framework. It first generates a structured analysis of a user by comparison between users. The generated structured analysis is then used to guide the generation of personalized summaries. To evaluate ComPSum, we propose AuthorMap, a fine-grained reference-free evaluation framework for personalized MDS. It evaluates the personalization of a system based on the authorship attribution between two personalized summaries generated for different users. For robust evaluation of personalized MDS, we construct PerMSum, a personalized MDS dataset in the review and news domain. We evaluate the performance of ComPSum, showing that it outperforms strong baselines. The code is available at https://github.com/leehaoyuan/comparative_personalize_sum.

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced reasoning capabilities in Large Language Models. However, adapting RLVR to multimodal domains suffers from a critical perception-reasoning decoupling. Existing paradigms, driven by text-centric outcome rewards, reasoning in language medium, inadvertently encourage models to bypass visual perception. We empirically validate this through blind experiments: paradoxically, state-of-the-art policies maintain or even improve performance even when visual inputs are entirely removed. This reveals that these models degenerate into blind reasoners, exploiting linguistic priors to generate plausible answers instead of attending to visual evidence. In response, we propose Thinking with Deltas, a framework driven by a Differential Visual Reasoning Policy (DVRP). DVRP introduces intrinsic supervision via visual triplets, comprising original, masked, and perturbed inputs. It optimizes the model to maximize reasoning divergence from masked inputs (enforcing visual sensitivity) while minimizing divergence from perturbed inputs (ensuring visual robustness). By aligning reasoning variations strictly with the Delta of visual information, DVRP inherently bolsters visual understanding capabilities and significantly outperforms state-of-the-art methods on both general and medical benchmarks, without requiring external annotations or auxiliary tools.

When an LLM judge scores an output, its score distribution retains uncertainty and disagreement information that is lost after scalar compression. We introduce JudgeMoE, a lightweight aggregator that assigns example-specific weights to cached judge score distributions and fuses them before computing a final score. A protocol study shows that score-range choice is unstable across judge–dataset settings and that soft scoring usually outperforms hard decoding. On the original 10-cell benchmark, JudgeMoE improves mean Spearman over uniform log pooling by +0.079. Applying the same configuration to six additional cells yields a +0.0393 mean gain over the strongest local single judge across 16 cells, with positive differences in 12/16 cells and a one-sided Wilcoxon signed-rank p=0.0091. Validation-based analyses further show that the preferred aggregation method depends on the task and judge pool.

Jailbreak attacks are usually compared by their attack success rate (ASR), with a higher ASR read as a stronger attack. But an ASR mixes an attack’s general strength with its fit to the particular model it was measured on, and it also depends on the judge that scored it. We separate these factors with a matched evaluation grid: five attacks on eight open-weight vision–language models, 200 HarmBench behaviours per cell, every turn scored by three judges (Qwen2.5-7B, Qwen2.5-72B and Llama-Guard-3), and each ASR decomposed into general strength, target susceptibility and target-specific fit. Attack rankings invert across targets: FigStep leads PAIR on InternVL2.5-8B (13.0% vs. 3.5%, paired p=4×10^-10) and trails it on Qwen2.5-VL-7B, in the same direction under all three judges. Across the grid, the target-specific component is statistically significant under every judge (p<10^-12). The judge matters as well: agreement between judges is low (Cohen’s 𝜅 between 0.03 and 0.29), and a larger judge with the same rubric changes which attack ranks first on three of eight targets. A higher ASR therefore shows that an attack succeeded more often on a given model under a given judge, but not whether its advantage is general or specific to that model. We release the protocol and ∼20,400 trajectories with per-turn scores under all three judges

Negotiation is a core component of social intelligence, requiring agents to balance strategic reasoning, cooperation, and social norms. Recent work shows that LLMs can engage in multi-turn negotiation, yet nearly all evaluations occur exclusively in English. We systematically isolate language effects across English and three Indic framings (Hindi, Punjabi, Gujarati) by holding game rules, model parameters, and incentives constant for multi-agent simulations for Ultimatum, Buy-Sell, and Resource Exchange games. Our results indicate that language choice is correlated to agreement and process dynamics, while final payoffs are statistically unchanged, and that this correlation persists across model architectures. Crucially, effects are task-contingent: Indic languages lower acceptance and longer negotiations in the Ultimatum Game, and higher trade volume in Resource Exchange. Further, they suggest that findings obtained under English-only conditions may not generalize to other linguistic settings. Even within one related language family, we observe task-contingent divergence from English, which cautions against treating English-only evaluation as representative. Whether this extends to typologically distant languages remains open.

Agentic Multimodal Large Language Models (MLLMs) are reshaping scientific discovery. They do more than answer questions: they read multimodal evidence, reason over scientific constraints, use tools, and close parts of the research loop. This survey explains how such systems are built for scientific discovery, moving from multimodal perception to training, inference, evaluation, and human-AI collaboration. Across this pipeline, scientific agents face bottlenecks that generic assistants often hide, including scarce expert annotations, evolving domain structures, complex reasoning, immature tool interfaces, safety and toxicity risks, and unfaithful results. We highlight design choices for turning agentic MLLMs into trustworthy, evidence-grounded collaborators. Curated resources are available at https://github.com/JianJinglin/awesome-agentic-AIScientists.

Conversational Intelligent Tutoring Systems (CITS) represent a convergence of artificial intelligence, learning sciences, and human-computer interaction, aiming to provide the adaptive and dialogic learning as expert teachers. In the past decades, the advancement of machine learning and natural language processing reshaped CITS’s design, implementation, and evaluation. In this survey, we analyze the evolution of computational paradigms in shifting pedagogical affordances, interaction styles, and methodological development. We frame CITS not merely as dialogue systems, but as proactive agents that reason about learner cognition, affect, and context while enacting pedagogical strategies grounded in educational theories. From domains, populations, and instructional goals, we identify key components, evaluation approaches, and unresolved challenges. Our paper highlights the promise and limitations of conversational tutoring in the generative AI era, offering a foundation for future research that seeks to maximize their pedagogical value.

Instruction-following machine translation (IF-MT) requires respecting prompt-level rules on terminology, formatting, and register. Rule compliance typically trades off against translation quality, a tension that general-purpose IF data augmentation methods do not address. We propose Reference-Grounded Data Curation, a two-phase pipeline that extracts every supervised constraint from a reference translation that already satisfies it, ensuring feasibility by construction. Phase 1 applies Instruction-Following Difficulty (IFD) scoring to retain the hardest-but-learnable instances from an English-Thai parallel pool. Phase 2 extracts constraints from each reference target and keeps only generations satisfying every constraint, yielding the 1.97M-record Grounded dataset. We fine-tune open-weight bases on Grounded to produce ChindaMT, a Thai-English translation family at 4B, 2B, and 0.8B parameters. Under length-controlled pairwise judging, ChindaMT outperforms or matches every same-size baseline at every tier on both plain translation and under explicit rules, reaching up to a 68.4% win rate against the strongest baseline. The recipe transfers cleanly across Qwen generations. We release model weights, the Grounded dataset, and evaluation suites.

Large language model (LLM) evaluation usually averages scores across many benchmark tasks, implicitly assuming broad task coverage is necessary for reliable general evaluation. This practice also implicitly treats all benchmark tasks as equally important for general evaluation. We revisit this assumption by treating benchmark results as a model-task score matrix under the performance prediction task. We find that a broad range of benchmarks are redundant with scores for 100 LLMs on 13 aggregate tasks and 523 subtasks. Specifically, we recovered model rankings with high rank correlation with the ALS matrix completion method after masking out several task scores. This means that no tasks dominated the recovery process, and both low-scoring and high-scoring tasks shared the same trends. Further analyses show that multiple-choice scores allow us to recover model rankings on free-form generation tasks. For general model ranking, free-form generation evaluation is not necessarily more informative than multiple-choice evaluation. These findings suggest that evaluating general-purpose LLMs can be accomplished with fewer tasks than the current common practice. Our analysis also estimates the number of models required to capture the overall trends in the benchmarks.

Despite the impressive reported performance of Large Language Models (LLMs) on medical Question-Answering (QA), evaluation still overrelies on multiple-choice and extractive settings primarily assessing textbook knowledge, with limited work on open-ended QA over patient records. We present a comprehensive study investigating LLM long-context (LC) comprehension on longitudinal, patient-centric Electronic Health Records (EHRs) across multiple datasets, models and metrics, covering extractive, multiple-choice, and open-ended generative QA under different context formulation and Retrieval Augmented Generation (RAG) settings. We show that medically fine-tuned models offer no measurable benefit over base counterparts, smaller models exhibit stronger memorization, and longitudinal supplementary notes aid generative QA. Reasoning models gain on complex multi-note QA under LLM Correctness but not stricter reference-aligned metrics, while hybrid RAG with reranking outperforms Full-Context (FC) prompting. A question-category analysis and multi-faceted metric evaluation further reveal where RAG and FC excel and struggle, with RAG falling behind in longitudinal reasoning, thus uncovering systematic failures pointing to the need for reasoning-appropriate metrics.

Video-guided machine translation (VMT) translates text by leveraging the context provided by the accompanying video. This additional contextual information helps improve translation quality by disambiguating ambiguous words or explicitly indicating attributes such as the colour or gender of objects and people. Prior works have extensively explored VMT; however, most approaches train models from scratch, which requires large amounts of data. Additionally, the majority of VMT research has focused on the English–Chinese language pair. In this paper, we explore training strategies for VMT using only a limited amount of data and extend the task to Indic languages, specifically Hindi, Marathi and Odia. We also propose a synthetic data framework that is effective for training VMT models in low-resource language settings. Our results demonstrate an improvement of +1.75 BLEU compared to baseline models.

Large language models (LLMs) perform strongly in machine translation, but their effectiveness as translation refiners depends on the draft source. We compare direct translation, self-refinement, and cross-system refinement on English-to-Chinese WMT test sets. Sockeye drafts obtain higher lexical-overlap scores but lower COMET than the stronger Qwen3 drafts, and receive larger gains after Qwen3 refinement. Additional controls qualify this pattern. Qwen3-0.6B drafts obtain the largest raw gain but lower final quality, while Qwen3-1.7B drafts start near Sockeye in COMET yet benefit less. Cross-family GLM-4-9B-Chat drafts also improve with Qwen3-8B, but achieve smaller gains and lower final quality than refined Sockeye drafts. Descriptive bucket, common-support, and regression analyses suggest that initial quality alone is an incomplete account of these differences. xCOMET-XL score and span diagnostics further characterize the contrast with Qwen3-8B self-refinement. We frame translation editability as a property of the evaluated draft–editor pairing, assessed jointly through quality gain, improvement probability, degradation risk, and final quality.

For very low-resource language pairs such as Bahnar-Vietnamese, the relative effectiveness of generic multilingual, language-pair-specific lexical, and task-adapted retrieval remains unclear. We study Bahnar-Vietnamese parallel-sentence retrieval (bitext retrieval) as a controlled candidate-ranking task and compare lexical, alignment-based, neural, and hybrid methods under a development-based selection protocol. The strongest off-the-shelf encoder, LaBSE, reaches 24.24% Recall@1, while symmetric IBM1 reaches 49.33%. Among four in-domain modern dense retrievers, development selection chooses DPR-XM, which reaches 47.43% Recall@1 and 72.21% Recall@10. XLM-R LoRA reaches 64.02% Recall@10, and the development-selected XLM-R-first IBM1 reranker gives the highest Recall@1 point estimate, at 51.62%, with an MRR of 0.5579. Paired tests do not establish a significant difference between the hybrid and IBM1 at the 0.05 level, although reranking substantially improves over its neural first stage. Under pair-specific training, LoRA projection with CSLS also performs strongly on Khmer-Vietnamese and Lao-Vietnamese, reaching 80.14% and 84.58% Recall@1. Overall, the results show complementary strengths between lexical alignment and lightweight neural adaptation in low-resource bitext retrieval. Data and code are available at https://github.com/NgThTy/translation-vn-bahna.

When an LLM judge only has to assign a three-way support label to a candidate answer given a reference, does asking it to decompose the answer into atomic claims help, and at what cost? We compare four single-call designs that share the judge model, the inputs, and the level of instruction detail: candidate-side atomic decomposition, a matched holistic rubric, reference-side decomposition that checks whether each reference claim is covered, and a bidirectional combination. Support labels are constructed from TruthfulQA, ASQA, and QAMPARI references (200 questions and 400 rows per dataset). All four designs are run with Opus-4.6, GPT-4.1, and Gemini Flash Lite; Sonnet-4.6 is added for the candidate-side and holistic designs. Candidate-side decomposition is weak where the label depends on completeness: the holistic rubric is more accurate on ASQA and QAMPARI for every judge while using fewer tokens. Reference-side decomposition is 12.5-21.3 points more accurate than holistic on ASQA for 28-29% more tokens, and stays near the holistic ceiling on QAMPARI at 63-69% more. On TruthfulQA misconceptions, candidate-side decomposition is competitive and significantly better for two judges. On a 60-row single-author subset whose labels are looser than strict reference completeness, candidate-side decomposition edges ahead of holistic for every judge, reversing the construction-label order. What a judge decomposes should follow what the label measures.

Hate speech classifiers often misclassify counter speech, including condemnation and quotation of hateful content, as hateful because they fail to distinguish propositional content from speaker stance. We identify this issue as content stance conflation and address it as a representation learning problem. We learn stance invariant representations of the same proposition by constructing contrastive pairs that share content but differ in stance. We implement this idea through factorized annotation, stance controlled augmentation, and Stance Contrastive Learning (SCL). On the Implicit Hate Corpus, stance variant augmentation reduces counter speech false positives by 60%, and SCL further improves cross domain F1 by up to 6.5 points on top of the augmented signal. On SBIC, where use mention conflation is less prominent in the training labels, SCL maintains a low counter speech false positive rate. Semantic similarity pairing on the same augmented data yields no cross domain gain over the model trained without contrastive learning (0.623 vs. 0.625), which identifies the pair construction signal rather than contrastive learning itself as the determining factor.

Speech-based Alzheimer’s disease (AD) detection increasingly relies on speech-enhanced and curated versions of the Pitt Corpus, where speech enhancement, sample selection, and demographic balancing are often treated as beneficial preprocessing steps. However, whether these transformations improve real-world AD detection or instead affect model generalization and prediction behavior remains unclear. In this work, we revisit the role of speech preprocessing and dataset curation across widely used benchmarks for speech-based AD detection. We evaluated the speech quality of different datasets, the cross-dataset generalization of multiple deep learning models under matched and mismatched enhancement settings, and the behavior of several recent large audio-language models (LALMs). Experimental results show that across multiple supervised speech models, speech-enhanced datasets often improve in-domain performance while reducing robustness in cross-domain evaluation. Matched enhancement between training and test data alleviates, but does not eliminate, this degradation. LALMs show a similar sensitivity: enhanced datasets induce stronger class imbalance and prediction shifts than unprocessed data. These results suggest that speech preprocessing and dataset curation can substantially influence downstream AD detection behavior, indicating that "cleaner" speech datasets are not necessarily more reliable for real-world AD detection.

Patient-facing medical LLMs and agents increasingly answer symptom questions before clinician contact, where the key safety question is what action the user should take next. We introduce CARE-Bench, a source-grounded benchmark that evaluates sequential patient-facing triage as a four-label per-turn current-action task. CARE-Bench contains 500 cases and 1,059 evaluated patient-disclosure prefixes reconstructed from medical dialogue, consultation, and follow-up-question sources. We evaluate 11 models on 269 held-out rounds under unprompted and minimally prompted open-ended protocols, using a fixed gpt-5.5 mapper to code each response into the four-label action space. Unprompted macro-F1 remains low, ranging from 31.2 to 50.4. Prompting improves 10 of 11 models, with prompted macro-F1 ranging from 46.9 to 63.4, but substantial threshold errors remain. Prompted models often recommend care before needed clarification is obtained; when the correct action was to ask for more information, only 33.5% of prompted outputs preserved the step. The persistence of these errors after prompting suggests that patient-facing triage is not a simple prompting problem and supports explicit evaluation of action timing before deployment.

Tang Dynasty poetry (618–907 CE) represents one of the most significant corpora in Chinese literature, yet no openly accessible, computationally-suitable resource exists with comprehensive author metadata. We present OpenQTS, a complete machine-readable edition of the Quan Tang Shi (全唐詩), containing 48,300 poems from 2,602 authors across 900 volumes. Unlike existing resources, OpenQTS provides systematic biographical metadata through algorithmic inference and authority file integration. Each poem receives a unique identifier, with proper handling of multi-part poems (15.5% of corpus) and collaborative works. All metadata includes provenance tracking, documenting whether information derives from Wikidata, biographical text inference, or historical markers.

Geo-spatial and geo-temporal reasoning require models to determine not only what information is present but also where and when it applies, which is critical for applications such as crisis response, environmental monitoring, and urban planning. This review surveys recent advances in vision–language models (VLMs) and large language models (LLMs) that address geographically and temporally grounded reasoning across tasks including image geolocation, spatiotemporal question answering, language-guided navigation, and Earth observation. We organize the literature through a structured analysis covering task families, input modalities, model architectures, and reasoning paradigms, highlighting how perception, retrieval, tool use, and intermediate reasoning steps contribute to system performance. The survey further synthesizes existing datasets and benchmarks, identifying common evaluation pitfalls such as spatial granularity mismatch, temporal drift, geographic bias, and potential data leakage. Finally, we outline key research challenges and opportunities for developing robust, verifiable, and time-aware geo-spatial AI systems while addressing broader concerns related to fairness, bias, and location privacy.

Large language models (LLMs) reason fluently but do not regulate their reasoning: they apply uniform scrutiny to every input, which leaves them vulnerable to adversarial and counterfactual prompts, while indiscriminate critique over-corrects answers that were already sound. We propose MetaCrit, a multi-agent framework grounded in Nelson and Narens’ metacognitive regulation theory that calibrates how much critique each task receives. MetaCrit separates regulation into four agents: an object-level generator, a monitoring agent that assesses response validity, a control agent that critiques logical soundness, and a meta-level synthesizer that reconciles their signals into a regulated response. Adaptivity here is input-conditioned intervention strength within a fixed pipeline: all four agents run on every input and what varies is the direction and magnitude of the correction they produce, not which stages execute. Across reasoning, safety, and bias benchmarks, MetaCrit improves truthfulness and logical soundness and reaches zero toxicity on BOLD and HONEST without a reasoning trade-off, whereas the same critique applied indiscriminately degrades performance. The cost is four calls per query, about one sixth of the cost of a dedicated reasoning model of similar accuracy. A writing study shows that MetaCrit is preferred for critical-thinking support, and its agents transfer to existing frameworks without architectural change. Code is available at https://github.com/Paparare/EduThink4AI.

Spatial reasoning, the ability to represent and reason about spatial relations, geometry, topology, and embodied environments, is increasingly vital for evaluating and deploying large language models (LLMs) and vision–language models (VLMs). While multimodal foundation models show strong performance on spatial benchmarks, navigation, and embodied planning, much of this stems from shallow pattern matching rather than grounded spatial understanding. This review offers a comprehensive, framework-driven analysis of spatial reasoning in LLMs and VLMs, covering tasks, representations, datasets, benchmarks, architectures, training, and failure modes. We also link spatial reasoning to embodied AI, emphasizing challenges in generalization, compositionality, and grounding. Our survey unifies diverse fields, NLP, vision, robotics, and cognitive science, to guide the development of truly spatially grounded foundation models.

Large language model agents have shown impressive capabilities in interactive decision-making tasks. These agents interact with environment through intermediate interfaces, such as predefined action spaces and interaction rules, which mediate the perception and action. However, mismatches often happen between the internal expectations of the agent regarding the influence of its issued actions and the actual state transitions in the environment, a phenomenon referred to as **agent-environment misalignment**. While prior work has invested substantially in improving agent strategies and environment design, the critical role of the interface still remains underexplored. In this work, we empirically demonstrate that agent-environment misalignment poses a significant bottleneck to agent performance. To mitigate this issue, we propose **ALIGN**, an Auto-Aligned Interface Generation framework that alleviates the misalignment by enriching the interface. Implemented as a lightweight wrapper, ALIGN-generated interface achieves the alignment without modifying either the agent logic or the environment code. Experiments across multiple domains including embodied tasks, web navigation and tool-use, achieve consistent performance improvements, with a 45.67% success rate improvement in ALFWorld.

Tone languages constitute over 50-70% of the world’s languages, but the vast majority are low-resource, lacking the large transcribed corpora needed for automatic tone classification. Existing datasets are typically collected at the sentence level, whereas field linguists require fine-grained syllable-level annotations. We propose ToneCL, a lightweight contrastive learning framework for few-shot syllable-level tone classification. We simulate low-resource conditions on Mandarin and Vietnamese, limiting labeled data to tens of examples per tone class. ToneCL is pretrained on unlabeled speech with augmentations that preserve tonal identity, then fine-tuned on few-shot examples. Experiments show our method consistently outperforms baselines, achieving 91.6% on six-speaker Mandarin at 10 shots. Cross-lingual transfer is also effective: pretraining on Vietnamese and fine-tuning on Mandarin reaches 91.0% accuracy at 10 shots. Ablation confirms that frequency band rejection is the most critical augmentation.

Hadamard preprocessing is widely used in large language model (LLM) quantization to reduce activation coherence and suppress peaks. Existing methods typically rely on random sign flips before the Hadamard transform, which do not use information from the target data distribution. We propose Greedy Hadamard Alignment (GHA), a simple training-free method to build a signed-permutation preconditioner from calibration data. GHA uses a pooled representative activation, a greedy search over flat targets, and an efficient alignment step that is analytically optimal once the target is fixed. We also study when one shared GHA preconditioner can transfer across LLM activations. GHA works best when activations are directionally aligned and stable across calibration sources. Overall, GHA improves post-quantization model quality over randomized Hadamard baselines, with the strongest gains in KV-cache quantization, more moderate gains for hidden states, and little compute overhead during inference.

Real-world documents are visually rich and structurally organized, with text, tables, figures, and captions interleaved across pages. Although retrieval-augmented generation (RAG) is widely used for document question answering, existing pipelines often treat such documents as fixed page- or chunk-level collections. This document-agnostic construction either dilutes evidence in dense pages or fragments semantically related layout elements. Moreover, cross-page questions often require iterative evidence exploration, but existing systems typically follow fixed retrieval paths that do not adapt to the evidence already found.

We propose DAMG-RAG, a Document-Aware Multi-Granularity RAG framework for visually rich document understanding. During ingestion, DAMG-RAG constructs layout-aware atomic retrieval units and links them to page- and section-level nodes, forming a multi-granularity document tree. During inference, a state-aware agentic workflow maintains explored regions and validated evidence, dynamically selecting exploration and navigation tools over this tree. On MMLongBench, DAMG-RAG obtains 64.7% accuracy with Claude Sonnet 4.6, yielding a 3.6-point gain over the best-performing agentic baseline in our experiments while using 18.8% of its token budget.

Recent LLM-for-Space systems address mission planning, scheduling, operations support, simulator control, and autonomy, but their evaluations use different task contracts, control settings, simulators, and success criteria. We introduce AstroAgentBench, a seven-family benchmark for executable space mission planning in the domains of scheduling, observation planning, constellation design, and relay support. For each case, an agent submits a planning artifact that is checked by an external verifier for schema, timing, geometry, resources, and mission value. Results report validity and normalized scores, with comparisons to task-specific solver references. Across five LLM agent systems and 35 held-out cases, the strongest systems approach or exceed solver-reference scores on several families, while weaker systems often fail to produce high-value valid plans and even strong systems lose quality on geometric, product-level, or design-heavy tasks. Trace analyses separate two failure points: task-contract misformulation and weak solution construction. Successful runs instead calibrate agent-written implementations against verifier feedback and adapt search to case-specific structure. Ablations show that procedure injection and memory accumulation help selectively, when they supply the missing formulation, calibration, or search support.

Large language models (LLMs) generally rely on sampling temperature to regulate the amount of randomness in their outputs. Mainstream methods for selecting temperature typically involve fixed, heuristic global settings and context or token-level calibration, rather than a geometrically interpretable representation of the communicative scenario. In this work, we introduce a simple and interpretable temperature selection framework, the **Scenario-Actor**, which adapts decoding to the inferred situational characteristics. Scenario-Actor extracts scenario-relevant structures from Transformer hidden states and determines the temperature from scenario embedding, a structured geometric representation of the communicative setting. Empirical results show that on Llama-3.1-8B-Instruct, Scenario-Actor achieves strong goal-completion performance in interactive social scenarios, significant gains on open-ended instruction following, and competitive performance on multi-turn conversation compared to multiple baselines.

Open-weight large language models face a low-cost white-box threat from representation engineering attacks. Attackers can estimate refusal directions and search for projection-matrix edits that suppress safety alignment while preserving general capabilities, within minutes on a single GPU and without gradient-based training. We propose Bait-and-Recover, a weight-level defense that places a bait adapter where attackers read activations and a paired recovery adapter at the subsequent layer. Trained via gradient routing, this decouples the observation path from the behavior path. By actively poisoning the residual signal used for measurement, Bait-and-Recover disrupts the attacker’s edit search, while the recovery layer restores clean downstream computation. Across four open-weight models, our defense raises the minimum refusal rate against white-box edit searches from 16.25% to 71.75% under a strict behavior-preservation budget (KL ≤ 0.10), with negligible impact on general benchmarks. By invalidating the core measurement assumption of these attacks, observation-path poisoning offers a practical complement to behavior-level safety training.

As text-to-motion (T2M) models move beyond short action captions toward real-world applications, they are increasingly expected to follow diverse instructions. However, existing evaluation protocols remain insufficient for systematically assessing instruction following, physical plausibility, and fine-grained controllability under such diverse language instructions. To address this gap, we introduce T2MBench, a benchmark with 1,025 prompts across four categories—Dynamics, Complexity, Interaction, and Accuracy—covering 14 representative T2M models and three evaluation dimensions: LLM-Based Evaluation, Multi-Factor Motion Evaluation, and Fine-grained Accuracy Evaluation. Our results reveal substantial gaps in current T2M models’ ability to consistently follow diverse language instructions while maintaining motion plausibility and fine-grained control.T2MBench establishes a reusable diagnostic benchmark for advancing research toward more reliable T2M generation under diverse language instructions. Data and materials are available at https://github.com/Courteous121/T2MBench-release.

Conversational multimodal emotion recognition MER requires reliable prediction when language acoustic or visual observations are missing or unreliable. Many missing-modality methods reconstruct absent inputs yet such recovery can be non-unique in dialogue context and nonverbal cues may conflict with the target utterance. To this end we propose CoRe-KD Complete-view Reference-guided Knowledge Distillation a state-anchored conflict-regularized complete-view distillation framework for robust conversational MER. A complete-view teacher provides structured references including prediction-level references fused states and modality-specific states. Complete-view State Anchoring CSA aligns incomplete-view student predictions and states with these references while Nonverbal Conflict Exposure NCE trains on target-preserving nonverbal conflict views to reduce donor-label bias. Experiments on IEMOCAP and MELD with CMU-MOSEI as a supplementary utterance-level check show consistent gains under fixed- and random-missing protocols. Comprehensive ablation studies and further analyses support the role of CSA and the complementary effect of NCE.

Sentiment analysis during political crises is difficult because language, context, and public reaction can change rapidly. This challenge is especially underexplored for low-resource languages such as Bangla. We introduce **UnrestSent200K**, a Bangla crisis sentiment dataset containing approximately 200K Facebook and YouTube comments from the July–August 2024 Bangladesh uprising. The dataset covers five event-aligned phases, ranging from early escalation and internet blackout to regime transition and a later flood crisis. Each comment is linked to its parent post, allowing evaluation both with and without discourse context. All comments are annotated through a fully human process involving 14 native Bangla-speaking annotators and senior validation. The annotation process achieves substantial agreement, with Cohen’s kappa of 0.73, Krippendorff’s alpha of 0.71, and 94.2% blind-audit agreement. We benchmark fine-tuned encoders, prompted LLMs, and LoRA-tuned LLMs. Results show that adding parent-post context consistently improves performance, while temporal shifts across crisis phases cause large performance drops. Strong LLMs perform well, but they still struggle with sarcasm, implicit political references, and meanings that depend on the event phase. UnrestSent200K provides a benchmark for studying context-aware and temporally robust sentiment analysis in low-resource crisis discourse.

Safety alignment of small language models (SLMs) typically incurs an alignment tax: benign prompts are refused along with truly harmful ones. The problem is sharper for Korean SLMs, where limited capacity meets a language whose intent signal is blurred by honorifics. We propose KoSCoPe, which combines a Hierarchical Safety Curriculum, a level-conditional adaptive margin (HARM-DPO), and Counterfactual Consistency Regularization, with a reasoning internalization schedule that recovers 1.5 to 3.2× inference speed. Across five Korean SLMs of 1.7B to 4B, KoSCoPe achieves the strongest Safety/Over-refusal trade-off among the baselines under our matched protocol and remains competitive with general-purpose models several times larger. Under paired-bootstrap tests, its Safety gains over the matched preference-optimization baselines survive multiple-comparison correction for all but the closest competitor, Cat-DPO, where the gain is smaller but directionally consistent. Preliminary cross-lingual validation on Japanese, Mandarin Chinese, and English suggests the same design can transfer beyond Korean, with larger gains where surface politeness markers are more salient.

Large language models show potential for scalable mental-health support by simulating Cognitive Behavioral Therapy (CBT) counselors. However, existing methods often rely on static cognitive profiles and omniscient single-agent simulation, failing to capture the dynamic, information-asymmetric nature of real therapy. We introduce CCD-CBT, a multi-agent framework that shifts CBT simulation along two axes: 1) from a static to a dynamically reconstructed Cognitive Conceptualization Diagram (CCD), updated by a dedicated Control Agent, and 2) from omniscient to information-asymmetric interaction, where the Therapist Agent must reason from inferred client states. We release CCDChat, a synthetic multi-turn CBT dataset generated under this framework. Evaluations with clinical scales and expert therapists show that models fine-tuned on CCDChat outperform strong baselines in both counseling fidelity and positive-affect enhancement, with ablations validating dynamic CCD guidance and diagnostic analysis supporting asymmetric interaction design. Our work offers a new paradigm for building theory-grounded, clinically-plausible conversational agents.

Legal citation in common-law systems depends not only on factual similarity, but also on the legal principle for which a precedent is invoked. However, existing benchmarks for legal citation retrieval use case facts, citation context or full judgments as inputs, where the governing legal principle is often missing or only implicitly expressed and entangled with broader context. As a result, models may retrieve precedents that are factually similar yet doctrinally irrelevant. This limitation is particularly consequential in Singapore, where the legal system has evolved independently: only domestic precedents are binding, while foreign authorities serve merely as persuasive references. Thus, we propose a new retrieval paradigm that ranks cited cases based on queries integrating case facts and explicit legal principle, inspired by the real-world legal reasoning workflows. To support this paradigm, we further introduce SG-LegalCite, a dataset of 100,890 case–principle pairs extracted from 8,523 Singapore Supreme Court judgments spanning 2000–2025. Experiments across 11 baselines demonstrate the effectiveness of our principle-augmented retrieval paradigm, showing that explicit legal principles provide strong discriminative signals for legal citation retrieval. Our dataset and code is at https://github.com/ShannonLeeYuehErn/SG-LegalCite

Fine-tuning has emerged as a widely adopted approach for adapting Large Language Models (LLMs) to a variety of downstream tasks. However, how it reshapes their internal mechanisms remains poorly understood. To address this, we investigate how fine-tuning alters internal representations in LLMs, including attention patterns and layer-wise activations, and examine whether these changes are linked to task-relevant components identified by Edge Attribution Patching (EAP) (e.g., attention heads and logit-level activations) that drive task performance. We find that EAP-identified components are concentrated within specific layers, indicating a degree of functional localisation in how models internalise task-specific behaviour. Notably, the distribution of these components across layers is largely uncorrelated with the layers undergoing the most substantial representational changes during fine-tuning. Furthermore, we observe that overlap in EAP-identified components across tasks does not translate into cross-task performance transfer if the tasks are different in nature (e.g., classification vs. generative tasks). More specifically, fine-tuning on one task can lead to a degradation of performance on another when the two tasks exhibit a high degree of overlap in their EAP-identified components.

Robust detection of AI-generated text under deployment conditions is challenging: distribution shifts across domains and generators, adversarial perturbations of the input surface, and the absence of target-domain labels for threshold calibration all degrade detectors that perform well in-domain. We present DeBERTa-ConPara, a deployment-oriented detector combining attack-aware Unicode preprocessing with a contextual transformer encoder trained over HC3 Plus, M4, MAGE and RAID. Our central finding is that preprocessing acts in opposite directions depending on where it is applied: normalising the training corpus deduplicates it, collapsing 35.4% of RAID rows into copies of their clean siblings and deleting the adversarial supervision, whereas normalising at inference is an effective defence. A factorial varying the two placements independently identifies raw training with normalised inference as the best configuration, reaching 99.61% AUROC, 99.01% TPR@5% FPR and 96.57% TPR@1% FPR on the official RAID hidden test, alongside 93.14% average balanced accuracy across HC3 Plus and MAGE under a fixed threshold. The gain is confined to two of twelve attack classes: homoglyph and zero-width-space insertion rise from 11.05% and 1.12% to 96.98%. The same signature reproduces in a zero-shot detector of different architecture, showing the effect belongs to the attacks rather than to our model. We additionally report two negative results: semantic-invariance augmentation through paraphrasing and supervised contrastive learning (ConPara) does not improve the best configuration, and the handcrafted feature-fusion branch is inert in distribution and harmful outside it.

Sight-singing is a fundamental yet challenging task in music education, where timely and fine-grained feedback is crucial for helping learners identify mistakes and improve their performance efficiently. However, existing automatic methods mainly focus on isolated subtasks such as alignment or scoring, and are unable to provide end-to-end diagnostic feedback with explicit identification of pitch, rhythm, and musical fluency errors. In this paper, we propose Cantus, a novel multimodal large language model framework for sight-singing assessment and exercise recommendation generation. Cantus jointly models vocal audio and symbolic musical representations, and leverages a pedagogically grounded error representation module together with a large language model to diagnose performance errors, generate structured assessment comments, and provide personalized exercise recommendations. To support this task, we construct a new dataset MIVO (MIDI-Voice) containing 1,003 real-world sight-singing samples with aligned MIDI notes and vocal recordings collected from authentic teaching scenarios. Experimental results demonstrate the effectiveness of Cantus in sight-singing assessment, achieving strong performance across multiple metrics. Human evaluation of feedback quality shows the potential of Cantus in supporting student-centered sight-singing learning.

General Retrieval-Augmented Generation (RAG) systems would suffer from performance degradation in multilingual scenarios due to the complexity of integrating different retrieval results from multilingual collections. In this paper, we propose CoSelector, a multi-agent collaborative framework for multilingual retrieval selection, which is optimized by the centralized training with decentralized execution. Under this paradigm, agents share the same goal but execute decisions using only local observations. CoSelector consists of a Selector agent and a Generator agent. The Selector dynamically filters and composes the most pertinent multilingual documents into a coherent context, while the Generator processes this context to understand and integrate information for answer generation. Across the 15 languages evaluated on MKQA and XOR-TyDi QA, CoSelector achieves competitive improvement in response language consistency, demonstrating its effectiveness in both linguistic alignment and the selection of retrieved results.

Long-horizon multimodal agents depend on external memory. However, similarity-based retrieval often surfaces stale, low-credibility, or conflicting items, which can trigger overconfident errors. We propose Multimodal Memory Agent (MMA), which assigns each retrieved memory item a dynamic reliability score by combining source credibility, temporal decay, and conflict-aware network consensus, and uses this signal to reweight evidence and abstain when support is insufficient. We also introduce MMA-Bench, a programmatically generated benchmark for belief dynamics with controlled speaker reliability and structured text–vision contradictions. Using this framework, we uncover the "Visual Placebo Effect," revealing how RAG-based agents inherit latent visual biases from foundation models. On FEVER, MMA matches baseline accuracy while reducing accuracy standard deviation by 35.2% and improving selective prediction utility; on LoCoMo, a safety-oriented configuration improves actionable accuracy and reduces wrong answers; on MMA-Bench, MMA reaches 41.18% Type-B accuracy in Vision mode, while the baseline collapses to 0.0% under the same protocol. Code is available at https://github.com/AIGeeksGroup/MMA.

Language models are increasingly expected to support users in stressful or uncertain situations, where warmth and empathy are desirable. While recent work links warmer models to lower factuality and reliability, we show that this conclusion is incomplete: prior studies often use warmth as a global persona and conflate tone effects with SFT data distribution shift. We revisit this issue through prompting, fine-tuning, factual robustness, and empathy evaluation, using an 11-dataset benchmark suite and EmoFact, a benchmark for factual accuracy under emotional context. Across three large language models, we find that warmth effects are task- and model-dependent, and that task-aligned adaptive tone can improve empathy while preserving reliability. We further train human-validated empathy evaluators to evaluate the output of our finetuned model. Our results suggest that models can be made more supportive in human relationships and social interactions without compromising overall reliability.

Knowledge graph question answering (KGQA) is a promising approach for mitigating LLM hallucination by grounding reasoning in structured and verifiable knowledge graphs. Existing approaches fall into two paradigms: retrieval-based methods utilize small specialized models, which are efficient but often produce unreachable paths and miss implicit constraints, while agent-based methods utilize large general models, which achieve stronger structural grounding at substantially higher cost. We propose RouterKGQA, a framework for specialized–general model collaboration, in which a specialized model generates reasoning paths and a general model performs KG-guided repair only when needed, improving performance at minimal cost. We further equip the specialized model with constraint-aware answer filtering, which reduces redundant answers. In addition, we design a more efficient general agent workflow, further lowering inference cost. Experimental results show that RouterKGQA outperforms the previous best by 3.57 points in F1 and 0.49 points in Hits@1 on average across benchmarks, while requiring only 1.15 LLM calls per question on WebQSP (1.23 on CWQ). Code and models are available at https://github.com/Oldcircle/RouterKGQA.

Memory-augmented assistants must answer questions from long-horizon conversations in which facts are not only missing, but also revised, corrected, or mutually inconsistent. Existing methods largely optimize retrieval, storage, or updating, and many controllers still treat failure primarily as insufficient evidence. We argue that conversational memory QA also requires an explicit model of inconsistency: when remembered claims disagree, a system should not simply retrieve more or guess, but may need to resolve versions, return a qualified answer, ask for clarification, or abstain. We introduce CoMMA, a transparent read-time controller with explicit conflict state for conversational memory QA. CoMMA extracts provenance-aware claims from retrieved memories, groups potentially incompatible claims into conflict clusters, infers lightweight question intent, and chooses among direct answer, qualified answer, clarification, and abstention. In an evaluation centered on the full LongMemEval-Oracle benchmark, a LongMemEval conflict-focused diagnostic slice, and FactConsolidation-20, CoMMA performs best on the main LongMemEval answer-quality comparison, is the only method with non-zero conflict F1 on the conflict-focused diagnostic slice, and ties the strongest write-time overwrite baseline on FactConsolidation-20. These results suggest that making conflict explicit at read time improves both decision quality and the auditability of memory-aware behavior.

Safety evaluations of large language models (LLMs) have largely focused on whether models refuse explicitly harmful requests. However, in high-stakes decision-support settings, seemingly legitimate requests may conceal missing information or hidden risk signals. We propose LogicLure, a benchmark for evaluating whether LLMs avoid premature decisions and actively seek evidence under such plausible scenarios with hidden risks. The benchmark spans eight domains and pairs persuasive visible narratives with hidden evidence maps accessible only through verification. Our evaluation measures not only the decision of models, but also whether they ask verification questions and whether their final decision is grounded in the revealed evidence. In our evaluation, the strongest model reaches only 26.4% evidence-grounded refusal despite 94.7% overall refusal. Furthermore, we analyze how scenario surface cues, explicit scrutiny setting, and prompt-induced stance shape these decisions. Together, these analyses characterize the key behavioral factors underlying LLM performance on LogicLure. The benchmark and code will be made publicly available.

Additive quantization enables extreme LLM compression while preserving lookup-table inference with free-form codebooks. At 2 bits per weight, however, greedy sequential initialisation can cause large quality losses that wider assignment search and PV-tuning do not fully erase. We show that this failure reflects an interaction between representational pressure and sequential codebook coupling. We use 𝜌=N/K^M as a structural pressure measure and propose OA-EM, a hard generalised-EM initialisation procedure using Hessian-weighted Mahalanobis distance to minimise output-aware reconstruction error. Across four models from 3B to 14B parameters, OA-EM consistently lowers matched post-PV perplexity. Controlled experiments at a fixed bitrate show that 𝜌 alone does not explain sensitivity to initialisation; the number and size of the additive codebooks also matter. A Euclidean-EM ablation and curvature diagnostics show that the gain is not explained by EM refinement alone: OA-EM reduces error in high-curvature directions. On Llama 3.2 3B, OA-EM also improves the measured quality-compute trade-off. Overall, the final quantized solution remains persistently dependent on initial codebook geometry. Code: https://github.com/kenno94-IK/aqlm-oaem.

We present FrenchNews-7, a cross-publisher France-based French-language news editorial desk classification benchmark combining a large multi-outlet corpus, a URL-derived seven-class taxonomy, and a fine-tuned CamemBERT classifier. Labels are assigned via a hybrid pipeline combining publisher URL slugs with LLM annotation for structurally ambiguous cases, audited through an inter-rater study (2 humans + 2 LLMs; pairwise κ ≥ 0.766, human–human κ = 0.806). We evaluate lexical, multilingual, and French-specific trained classifiers under both in-distribution and held-out-publisher settings, with additional comparison against zero-shot LLM baselines (GPT-OSS-120B, Mistral Small 3.2, Llama-3.3-70B) on the held-out pool. The strongest model, CamemBERT-base on full article text, outperforms headline-only input, generalizes to unseen outlets, and exceeds all three zero-shot LLM baselines on overall recall (0.799), with the gap concentrated in the ambiguous editorial-boundary categories Économie and Société. Cross-publisher evaluation reveals uneven boundary stability: Sport, Culture Loisirs, and International transfer cleanly, while Économie (recall = 0.517) is statistically indistinguishable from blinded human agreement on the same articles (0.55, n = 40), and Société (precision = 0.577) absorbs boundary ambiguity, both suggesting editorial conventions rather than recoverable classifier headroom. The fine-tuned CamemBERT-base model is released on Hugging Face (https://huggingface.co/LeFrenchNewsLab/camembert-base-frenchnews7) and the labeled manifest on (https://huggingface.co/datasets/LeFrenchNewsLab/frenchnews-7), alongside reference collection scripts to assist re-fetching and a reliability-tier guidance table.

We introduce SweSpecQA, to our knowledge, the first Swedish benchmark for specialized institutional and professional QA with paired text and full-page PDF evaluation. It contains 1,500 questions from public Swedish examination and certification materials across six domains: EU aptitude, boating, antibiotic stewardship, interpreter licensing, physician licensing, and pharmacist licensing. Of these, 234 questions require an original visual or layout-dependent source element; the remaining 1,266 are text-eligible. All benchmark questions are drawn directly from Swedish-language source materials, avoiding translation artifacts in multilingual adaptations of English benchmarks. We evaluate 17 models under a closed-book protocol, reporting accuracy by domain, exam subtype, visual/layout dependence, modality, and prompting strategy, as well as whether models follow the requested answer format. On the text-eligible subset, every multimodal model performs worse with full source-page input than extracted text; PDF penalties range from 4.0 to 38.2 points under direct prompting and from 1.7 to 36.4 points under chain-of-thought prompting. Reasoning prompts often improve PDF performance for several open-weight VLMs, while sometimes reducing clean-text accuracy. These results show that document-native evaluation exposes failures hidden by text-only evaluation and that preprocessing remains important for reliable Swedish exam-style QA.

Small language models (SLMs) are efficient and scalable, but their multilingual capabilities degrade severely at sub-billion scales, especially for Southeast Asian (SEA) languages. We introduce DuDi, a dual-signal multilingual distillation framework that combines an online sequence-level signal with off-policy and on-policy token-level signals. DuDi further uses a cross-lingual verbalizer to refine teacher feedback and improve teacher-student transferability in multilingual settings. Experiments on SEA-HELM across multiple model families, scales, and teacher-student settings show that DuDi consistently outperforms competitive distillation baselines. Ablations and analyses confirm that sequence-level optimization, token-level supervision, and cross-lingual verbalization provide complementary and transferable learning signals for multilingual SLMs.

Frontier LLMs perform well in Western contexts, but remain poorly tested on underrepresented cultures such as those in Southeast Asia (SEA). Existing NLI benchmarks are largely Western-centric, translation-derived, or monolingual, limiting their ability to measure culturally grounded reasoning. We introduce SEA-NLI, a native, culturally grounded NLI benchmark covering eight SEA countries in English and native regional languages, verified by native speakers. Across 17 encoder and decoder models, we observe a low performance from all models, especially for knowledge-intensive categories such as Languages and Science and Technology. Our analysis shows that failure cases mainly stem from missing SEA cultural knowledge: SEA-adapted models and culture-aware prompting improve performance, while CoT prompting offers limited gains.

Large Language Model (LLM) personalization is challenging because user preferences are rarely stated explicitly; instead, they emerge as implicit, multifaceted signals in users’ past interaction histories. Existing methods often compress user histories into a single representation or retrieve and inject past records without explicitly linking them to supporting evidence, which can lead to hallucinated or overly generic outputs. In this work, we propose Multi-Aspect PersonaLized Modeling with Evidence (MAPLE). MAPLE structures implicit user preferences through a MULTI-ASPECT PREFERENCE PROFILE and connects them to target-specific grounding evidence through a PERSONALIZED EVIDENCE PLAN. By selecting only generation points that are both relevant to the user and supported by available evidence, MAPLE improves personalization and grounding faithfulness simultaneously. Experiments on the Games and Books domains with two backbone LLMs show that MAPLE consistently outperforms strong baselines and generalizes robustly under limited-history and unseen-user settings.

While in-context machine translation approaches lead to notable progress for low-resource languages, it is not yet clear what kinds of linguistic information in prompts can consistently and reliably lead to performance gains. We evaluate whether English semantic representations, specifically Abstract Meaning Representation (AMR) and its extension Uniform Meaning Representation (UMR), can improve LLM-based MT via prompting. Across 12 languages, 7 models from 5 families, and multiple prompting strategies, we find that conclusions about their effectiveness are highly sensitive to the setting. We further observe that prompts using UMR tend to outperform those using AMR, particularly in few-shot scenarios and for certain models. These results highlight the need for comprehensive evaluation for in-context learning work.

Sentence encoders are foundational to a wide range of NLP applications, yet they are predominantly evaluated through performance on downstream tasks. Such task-specific evaluations provide limited insight into whether the resulting sentence representations capture fundamental compositional relationships in a task-independent manner. To address this gap, we introduce a set-theoretic framework for evaluating the compositional properties of sentence embeddings. Drawing on classical set theory, we define six evaluation criteria based on three core set-like text operations: TextOverlap, TextDifference, and TextUnion. We use these criteria to systematically evaluate 19 sentence encoders, including 7 classical models and 12 Large Language Model (LLM)-based encoders. Our results reveal substantial differences in their compositional behavior: notably, SBERT exhibits the strongest overall alignment with the proposed set-theoretic criteria, outperforming even recent LLM-based encoders. We further examine whether these intrinsic compositional properties are associated with downstream performance across five NLP tasks. Among them, summarization exhibits the strongest correlation with the TextOverlap and TextDifference criteria, suggesting that our framework may also provide useful signals about downstream capability without relying exclusively on task-specific evaluation. Finally, we release a benchmark comprising ~192K samples to support systematic evaluation and future research on the compositional properties of sentence embeddings.

Reasoning Vision-Language Models (VLMs) achieve strong performance on complex multimodal tasks, but reliable real-world application requires handling visual inputs that are messier than clean, curated benchmarks. Existing works mainly evaluate such reliability of VLMs through input corruptions, such as noise, blur and weather effects, which make visual evidence harder to perceive. This leaves a critical reliability failure mode underexplored: a model may perceive the evidence correctly, yet reason from plausible but irrelevant and distracting evidence and propagate this mistake to its final answer. To address this gap, we introduce **Distract-Bench**, a benchmark for evaluating VLM robustness to **semantic visual distractions**, defined as meaningful but task-irrelevant visual cues added to inputs while preserving the ground-truth answer. We comprehensively evaluate eight leading open-source and two closed-source VLMs across conventional vision corruptions and Distract-Bench. Our results show that Distract-Bench exposes a robustness failure distinct from vision corruptions: reasoning VLMs largely track their non-reasoning base models under perceptual degradation, but show consistently lower robustness to semantic distractions. Further analysis shows that these distractions often enter the reasoning process of VLMs, are treated as evidence, and lead to incorrect answers. Together, these findings reframe robustness evaluation for reasoning VLMs, shifting the focus from degraded perception to distractions for reliable real-world visual reasoning. Our data and code are available at _https://github.com/Yizheng-Sun/Distract-Bench_.

Multimodal Large Language Models (MLLMs) struggle with continual learning, often suffering from catastrophic forgetting when adapting to sequential tasks. We introduce a routing-based architecture that integrates new capabilities while robustly preserving foundational knowledge. While Multi-Task Learning (MTL) is a strong performance reference, its training cost grows linearly with the number of tasks. In contrast, our method maintains fixed per-task training data and compute requirements regardless of the task sequence length. Across models ranging from 2B to 8B parameters, we demonstrate that our routing approach performs on par with MTL while retaining the training efficiency of sequential fine-tuning. Beyond merely mitigating forgetting, we observe that token-level routing facilitates cross-modal transfer, leveraging knowledge from one modality to bolster performance in another. We also show that routing remains robust even with large expert pools and effectively capitalizes on task relatedness. Finally, we show that our method scales favorably, with larger models exhibiting minimal degradation compared to fully specialized fine-tuning.

When a language model ignores retrieved evidence and answers from memory instead, this is not a knowledge failure, the correct information is present. It is a routing failure; the model commits to its parametric answer despite the evidence signal. We show this distinction is mechanistically meaningful, localizable, and correctable at inference time without training data or weight modification.

A single prefill-margin computation detects routing failures (AUC ≥ 0.93 across five model dataset pairs) and gates a format-appropriate logit-bias correction. On MedQA multi-option conflicts, this achieves 77.5% restoration at zero harm, exceeding the strongest prompting baseline by 67.5 pp and matching domain-matched ITI without training data. On open-ended PopQA (3,141 items, three models), balanced alias promotion achieves 80-88% restoration at near-zero harm, Pareto-dominating prompt-only baselines in five of six cross-model conditions. The central mechanistic finding is that effective late-layer intervention sites are structurally distinct from the causal commitment locus (a five-head cluster, joint flip rate 88.6%, p < 10^-9), suggesting that steering can succeed through residual-stream write access without exactly identifying the full decision circuit.

Prompt-based prediction pipelines suggest that large language models (LLMs) perform poorly on time-series classification, raising doubts about whether they encode meaningful temporal structure. We show that this conclusion reflects limitations of prompt-based generation rather than the model’s representational capacity by directly comparing prompt outputs with linear probes over the same internal representations while controlling for important confounding factors. While prompting performs near chance, linear probes improve average F1 from 0.15–0.26 to 0.61–0.67, often matching or exceeding specialized state-of-the-art (SoTA) time-series models. Layer-wise analyses further show that class-discriminative time-series information emerges in early transformer layers and is amplified with multimodal inputs and the presence of pretrained weights. Together, these results demonstrate a systematic mismatch between what LLMs internally represent and what prompt-based prediction pipelines reveal, leading current prompt-based prediction pipelines to underestimate the degree to which these models function as useful time-series feature extractors.

Medical vision-language models (VLMs) commit clinical errors when generating chest X-ray reports: they fabricate findings that are not present in the image, miss important ones, or locate them incorrectly. We steer the error composition of generated reports without weight updates by decoding-time residual steering on a per-token sparse autoencoder (SAE) basis: Top-K SAEs on late layers, per-error-type causal feature screening against clinical errors, then combined suppress/boost intervention at inference time. On the MIMIC-CXR test split, our inference-only method improves the quality of generated reports for three radiology VLMs (RadVLM, LLaVA-Rad, and CheXOne), with relative improvements of +5.4%, +7.2%, and +17.0% in the clinical composite metric, and statistically significant GREEN gains on all backbones. The gain is an explicit, controllable trade-off rather than uniform error reduction: steering shifts models toward more complete reporting, substantially fewer missing findings, at the cost of some additional false findings, and we analyse and quantify this redistribution per error type. A cross-model feature alignment shows that the quality-promoting (boost) directions have functionally similar error-type profiles across architectures, whereas hallucination-linkedm (suppress) directions are model-specific. Census-motivated steering should therefore treat suppression per-backbone, rather than sharing a universal suppress list. The same recipe applied zero-shot to IU-Xray (GREEN +7.7% rel., on RadVLM) suggests that the identified features are properties of the model rather than of the training corpus. The causal feature sets and an interactive feature dashboard are shared.

This position paper argues that researchers should not only study how human biases enter large language models (LLMs), but actively leverage those biases as diagnostic tools to surface society’s "unwritten code"—the tacit norms and hidden heuristics that govern high-stakes decisions yet resist explicit articulation. We present a case study in peer review: a framework that iteratively pushes LLMs to generate self-consistent hypotheses explaining score differences across 26,731 submissions to 46 conferences. LLMs’ normative priors (cues extracted by self-talk) emphasizing theoretical rigor are systematically updated toward posteriors (cues actually used in the judgment) emphasizing storytelling and contextualization—a finding cross-validated by independent human survey evidence. Human reviewers explicitly reward prior-aligned aspects (r=0.49) but rarely articulate the storytelling posteriors (r=-0.14), despite implicitly rewarding them. These findings generalize to out-of-sample judgment and different models. We argue this framework applies to hiring, admissions, and criminal justice, offering a scalable path to making hidden evaluative codes visible for public scrutiny and equitable reform.

Estimating model performance without labeled data is important for deploying NLP systems across diverse domains. Prior work has proposed proxy measures based on distribution shift or predicted correctness, but it remains unclear whether these methods reliably preserve the ranking of domains, which is critical for model selection and risk triage. We study this problem from an ordinal perspective using a two-step framework that pairs a base classifier with an error predictor to estimate instance-level failures and disparities and aggregate them into label-free performance estimates. Across multiple sample sizes from the GeoOLID and Amazon Reviews datasets, large language model–based error predictors substantially outperform drift-based baselines in 𝜌 and 𝜏 rank correlations. We further show that lower absolute estimation error (RMSE) does not necessarily imply better ranking, highlighting the importance of evaluating ordinal consistency directly.

WARNING: This paper contains offensive and hateful speech in the appendix.

Persona-driven generations (PDGs) have seen prolific use in research and industry applications, where a large language model (LLM) takes on a “persona” while completing some task. While persona expressed through free-form text (like dialogue) has substantial work investigating stability or consistency, relatively, persona expressed in non-text-heavy outputs (like in multiple-choice question answering, or MCQA) is often overlooked. We work to address this gap, seeking to understand the instability of LLM PDGs in MCQA tasks. We develop three metrics investigating the performance, outcome, and question correctness stability, evaluating three distinct dimensions. Using these metrics, we find that instability varies consistently between model families and model size, and across question domains, with math/commonsense questions leading to greater instability. We also find task prompt format introduces more prediction instability than other hyperparameters, like temperature. Finally, we find that instability is related to task accuracy, and using our instability metrics, find different experimental settings that result in different best and worst personas for tasks, despite their similarity. This reveals the importance of checking hyperparameter instability in PDGs.

Recent work has explored the use of large language models (LLMs) to generate tutoring responses in mathematics, yet it remains unclear how closely their instructional behavior aligns with expert human practice. We analyze a dataset of math remediation dialogues in which expert tutors, novice tutors, and seven LLMs of varying sizes, comprising both open-weight and commercial models, respond to the same student errors. We examine instructional strategies and linguistic characteristics of tutoring responses, including uptake (restating and revoicing), pressing for accuracy and reasoning, lexical diversity, readability, politeness, and agency. We find that expert tutors produce higher-quality responses than novices, and that larger LLMs generally receive higher pedagogical quality ratings than smaller models, approaching expert performance on average. However, LLMs exhibit systematic differences in their instructional profiles: they underuse discursive strategies characteristic of expert tutors while generating longer, more lexically diverse, and more polite responses. Regression analyses show that pressing for accuracy and reasoning, restating and revoicing, and lexical diversity, are positively associated with perceived pedagogical quality, whereas higher levels of agentic and polite language are negatively associated. These findings highlight the importance of analyzing instructional strategies and linguistic characteristics when evaluating tutoring responses across human tutors and intelligent tutoring systems.

Traditional lossless text compression preserves every byte, but its gains on natural language are often modest. We study lossy semantic text compression, where the encoder strategically deletes parts of the text and a large language model (LLM) reconstructs the original from the retained skeleton. We benchmark a progression of deletion strategies, from character-level and word-level rules driven by word length or corpus frequency to surprisal-based and hybrid rules that require a neural encoder, and reach three findings. First, deletion by corpus word frequency is a strong low-cost baseline: a static frequency lookup remains competitive with far more expensive semantic methods. Second, semantic and hybrid methods gain most at mild-to-moderate compression, whereas at the lowest retention rate frequency-based deletion matches them and keeps more named entities. Third, QLoRA fine-tuning yields a compact local decoder that, once adapted, is competitive with zero-shot Gemini 2.0 Flash. Additional English and Chinese experiments show that the framework transfers across domains, while the best deletion rule remains dataset-dependent.

While Large Language Model (LLM) based agents excel at complex tasks, their performance in open-ended scenarios is often constrained by isolated operation and reliance on static databases, missing the dynamic knowledge exchange of human dialogue. To bridge this gap, we propose the INteractive Memory Sharing (INMS) framework, an asynchronous interaction paradigm for multi-agent systems. By integrating real-time memory filtering, storage, and retrieval, INMS establishes a shared conversational memory pool. This enables continuous, dialogue-like memory sharing among agents, promoting collective self-enhancement and dynamically refining the retrieval mediator based on interaction history. Extensive experiments across three datasets demonstrate that INMS improves agent performance by effectively modeling multi-agent interaction and collective knowledge sharing.

Agentic task-oriented dialogue (TOD) requires systems to track concurrent goals, dependencies, and long-horizon state. We examine goal-lifecycle recovery from fixed dialogue trajectories. ATOD contains 1,000 synthetic dialogues annotated for six advanced-TOD properties, and ATOD-Eval defines metrics for dependency-sensitive completion, memory recall, and proactivity. We implement a symbolic–vector memory evaluator for ATOD-Eval. Among five prompt-only predictors and six matched-backbone memory baselines, our evaluator is the only configuration above 90% in both goal detection F1 and conditional status accuracy on medium dialogues. On complex dialogues, no baseline exceeds it on both metrics; it has the highest conditional status accuracy within the memory-based block and the lowest measured per-turn latency. These experiments assess lifecycle tracking rather than interactive agent task success. A cross-family judge swap and a manual audit with 94.0% agreement provide initial checks on measurement reliability. Code and data will be released at https://github.com/amazon-science/ATOD.

Role-Based Access Control (RBAC) is fundamental in secure systems handling healthcare, financial, and government data. Prior work on LLM safety under role constraints has largely relied on fine-tuning or prompting-based enforcement. In contrast, we take a mechanistic view and probe internal representations to identify signals associated with roles, permissions, and refusal behavior. We evaluate this approach across three datasets, database- and department-separated splits, varying policy complexity levels, and adversarial prompts. On the structured Text-to-SQL datasets, simple probes achieve strong performance, reaching average macro-F1 scores of .913 on BIRD and .926 on Spider, while remaining more stable than prompting as policies grow longer. However, probing does not perform equally well in all settings: it underperforms prompting in most OrgAccess comparisons and can degrade substantially under adversarial instructions. Overall, our results show that access-control information can be strongly represented in LLM hidden states, but its usefulness depends on the model, domain, and prompt.

Multimodal emotion recognition (MER) in conversation is inherently ambiguous, as linguistic, acoustic, and visual cues may support multiple plausible interpretations. Existing approaches often treat ambiguity as prediction uncertainty rather than modeling its structured sources. We propose UAF, an ambiguity-aware framework for MER built around a single unified ambiguity signal. UAF estimates a normalized per-sample Model-based Ambiguity Degree (MAD) from the predicted emotion distribution by measuring its proximity to the maximally ambiguous uniform distribution, and uses it as a shared control signal rather than a post-hoc uncertainty score. This single scalar coordinates three parts of learning: it aligns representation radius with ambiguity, emphasizes ambiguous samples during optimization, and scales OT-based perturbation consistency. The fused evidence is organized into context, speaker, temporal, cross-modal, and residual subspaces that serve as the substrate on which this signal operates. Experiments on IEMOCAP, CMU-MOSEI, and MELD show that UAF yields consistently better calibration and high-ambiguity reliability alongside competitive accuracy and weighted-F1, showing that a single ambiguity signal can be an effective shared training signal for reliable MER

Large language models are increasingly used as math tutors, raising two questions: can a single LLM detect a student’s error, diagnose the underlying misconception, and remediate it through multi-turn dialogue—and can we trust the numbers that report how well it does? We address both with MisconceptDialogue, a fully automated framework that runs the full pipeline on five open-weight LLM tutors (7B–14B) interacting with LLM-simulated students over 5,000+ dialogues—a deliberately simulated protocol (paired conditions, a wrong-diagnosis control unsafe with real learners, scale) whose results do not establish real-student learning. On measurement, our headline finding is uncomfortable for the field: benchmark-style exact-match scoring common in standard math NLP—a lower-bound stress test—reports 42.1% measured Remediation Success Rate on identical dialogues where a human-calibrated LLM-as-judge reports 81.3%, a 39.2 percentage-point spread larger than the combined effects of tutor model, teaching strategy, and supplied diagnostic information. This is not a single-judge artifact: four independent LLM judges and 200 human annotations converge to within a few points (κ up to 0.91). On capability, the verdict is equally striking: the score barely changes whether we tell the tutor the student’s actual misconception or a deliberately swapped one—under the tested diagnosis-presentation format, these tutors show little evidence of reliably conditioning on the diagnostic information they are handed. For LLM tutoring evaluation to mature, multi-criterion scoring with multi-judge convergence and human anchoring should become standard practice—and the measurement question deserves as much scrutiny as the capability question it claims to answer.

How do interpretable internal features that implement a learned capability emerge during training? We present a mechanistic interpretability study of 3-digit addition in a 19M-parameter transformer trained from scratch, using the grokking transition as a natural microscope for feature genesis. Our framework, Causal Genesis Attribution (CGA), combines checkpoint-resolved training dynamics, Sparse Autoencoder (SAE) feature analysis, and counterfactual training interventions to trace arithmetic and carry features from emergence to causal function. We find layer-4 SAE features that are silent on non-arithmetic text, selective for the arithmetic probes, and co-emerge with validation accuracy at step 500 (near-zero to 85%). Zero-ablating the top arithmetic features drops exact-match accuracy from 100% to 0%, showing that these directions are necessary for the learned addition algorithm. A four-arm intervention study reveals what sustains these features: removing arithmetic data collapses feature activation by 81%; matched numeric-noise replacement also fails, ruling out surface token statistics; and poisoning answers by +137 hyperactivates features by 27% while making the model produce systematically shifted outputs. Across four seeds, the genesis detector fires consistently, with detection accuracy stable at 85.6% ± 1.0%. Together, these results provide a mechanistic data–feature–capability chain for an emergent arithmetic reasoning ability.

LLM benchmarks often pool rows from several sources (judges, retrieval configurations, training frameworks) and report one headline score. When label rates differ by source, a detector can earn that score by recognizing a row’s source, not its quality, and such a score is no promise about a new source. Our audit reports three AUCs of one detector: given features plus source (AUCfull, the headline), features alone, and source alone. Their ratio L=AUCsource/AUCfull shows how close the source alone comes to the headline. The precondition: L is read only above a default floor, AUCfull≥ 0.65, because two near-chance AUCs give a ratio near 1 although nothing leaks. On four RAGBench configurations (adherence rates 40.2% to 94.2%) the headline is 0.843 and the configuration label alone reaches 0.811 (L=0.96), above every per-query feature. When five LLM judges grade the same hard-math answers, L=0.84 looks similar, but the headline (0.594) fails the floor: judge identity predicts nothing. The floor, not L, separates the two panels. Six further audits with the floor and a 0.90 band on L frozen add the other outcomes, including a floor-passing panel whose source carries nothing.

Patch retrieval, the task of finding the commit that fixes a known vulnerability, is the foundation of vulnerability management workflows, yet 60% to 63% of CVEs in the major advisory databases lack a patch link. We present PatchHolmes, a two-phase patch retrieval system that pairs a hybrid first-stage retriever with an agentic second-stage inspection loop. Unlike pointwise prior work that scores each candidate independently, the Phase 2 agent reads the top-100 listwise: it sees the full candidate list at once and selectively reads 3 to 10 commits through four budgeted tools before submitting a single best commit. On GitHubAD, PatchHolmes beats the pointwise binary classifier Favia by 25.34% Recall@1 and the retrieve-and-CoT baseline IRCoT by 31.40%, at one agent conversation per CVE versus Favia’s ten; with the candidate set held identical, the agent adds 27.32% Recall@1 over taking the retriever’s top candidate, and the same agent, transferred unchanged to PatchFinder_top10, lifts Recall@1 from PatchFinder’s own top-1 pick (24.28%) to 39.86%. Swapping the LLM backbone within the Qwen family changes Recall@1 by under 1%, and a second model family (gpt-oss) stays far above the no-agent floor, so the gain comes from the listwise agent loop; the entire system runs on a frozen open-weight model over a local Git repository, without fine-tuning or external search APIs.

Constrained decoding is widely used to make large language models produce structured outputs that satisfy schemas such as JSON. Existing work mainly treats schemas as structural constraints, overlooking that schema-key tokens also enter the autoregressive context and may guide generation. To the best of our knowledge, we present the first systematic study of schema keys as an implicit instruction channel under constrained decoding. We formulate structured generation as a multi-channel instruction problem, where task signals can be placed in prompts, schema keys, or both. We further provide a projection-aware analysis that gives a sufficient condition under which an unconstrained expected-score advantage of an instructional key is preserved after grammar projection. Experiments on GSM8K and Math500 across seven language models show that changing only schema-key wording can substantially affect accuracy, with both positive and negative effects across models. Prompt-level and schema-level instructions also interact non-additively. The evidence is substantially stronger on GSM8K than on Math500. Our findings show that schema design is not merely output formatting, but part of instruction specification in structured generation.

Chain-of-thought (CoT) faithfulness is measured almost exclusively in single-turn settings, leaving open how the reasoning-trace-to-answer relationship evolves as conversations lengthen. We introduce turn-level CoT counterfactual deletion: at every assistant turn we regenerate the answer at five truncation levels of the current-turn thinking block under greedy decoding, and label a turn anchored—a descriptive label for causal redundancy of the trace, not a mechanistic claim—when all five regenerations agree on the same answer. Across 67 multi-turn sharded-GSM8K conversations from DeepSeek-R1-Distill-Qwen-7B, the anchored fraction grows 3.9× from short (< 10 turns, 7.2%) to long (> 20 turns, 28.4%). Strict early < mid < late monotone ordering holds in five of seven reasoning models from three base families and four post-training pipelines (binomial sign test, p=0.002). A supplementary pooled GEE across all seven models converges on this direction (β=1.12, p=5.1×10⁻⁸, n=2,838 turn-level rows). A surface-feature predictor flags anchored turns at AUC= 0.71 (+0.07 over a length-only baseline) without additional generation, providing a cheap online signal for CoT monitors calibrated on short conversations. Single-turn CoT benchmarks, by construction, cannot detect this gradient.

Value alignment of Large Language Models (LLMs) has been shown to be culturally biased toward Western norms. This results in the mishandling of local values in multilingual societies such as Sri Lanka that have their unique cultural dynamics. Existing benchmarks overlook Sri Lankan-contextualized values in its official language Sinhala, hindering culturally sensitive evaluation and fine-tuning. To bridge this gap, we propose LKValues, the first survey-grounded resource suite for Sri Lankan value alignment. From a trilingual survey of 205 respondents, blending adapted global frameworks and LLM-elicited local constructs, we derive 40 majority-endorsed societal values. Using these values, we construct LKvaluesIT, a Sinhala-English news-derived instruction corpus containing 150K scenario-based instances, and LKvaluesBench, a value-sensitive evaluation benchmark of 1,000 instances. We evaluate a set of proprietary and open-weight LLMs with LKvaluesBench. We fine-tune three open-weight base models (Qwen3.5-4B-Base, Qwen3.5-9B-Base, and Aya-Expanse-8B-Base). Our experiments show that newer and larger LLMs still exhibit low-resource and cultural value-alignment gaps. LKValues fine-tuning improves Qwen-family models in English and Sinhala, reducing invalid outputs and cross-lingual disparities, though gains remain model-family dependent. These highlight LKValues’ efficacy in embedding Sri Lankan values, offering a replicable pipeline for low-resource, country-specific pluralist value alignment. The datasets are publicly available at https://github.com/NextME14/LKValues.

Large language models routinely produce numerical hallucinations—plausible but incorrect quantities, percentages, and statistics—yet existing benchmarks focus almost exclusively on entity-level factuality. We introduce EcoStatKG, a knowledge graph (KG) of 50,746 statistical triples and 306,231 structural triples from 6,871 Wikipedia articles on environmental sustainability, and NumHallu, a 500-question numerical benchmark grounded in held-out KG topics. We use these resources to diagnose how knowledge-base coverage affects eight retrieval strategies across six LLMs. In the full-index, oracle-coverage condition, simple cosine retrieval reaches 0.989 Numerical Exact Match (NEM), statistically indistinguishable from Graph RAG (0.992; paired-bootstrap p=0.14). In the deliberately constructed held-out condition, which removes answer-bearing triples, retrieval methods fall below the zero-shot baseline (0.506 NEM), exposing the risk that topically related numerical distractors can override parametric knowledge. Similarity-threshold filtering restores KG Cosine to near-baseline performance (0.511), rather than establishing a new state-of-the-art. A bounded clinical-domain proof of concept reproduces the coverage and distractor patterns. The contribution is a diagnostic study and resource: its results clarify when numerical retrieval helps, when missing coverage can make it harmful, and why confidence gating is best viewed as damage control.

Mental health stigma has profoundly harmful impacts but its complexity makes it difficult to evaluate. Stigma may involve explicit derogation, but also subtler forms of blame, fear, paternalistic pity, social distancing, structural exclusion, and discrimination. We introduce a theory-grounded benchmark for automatic evaluation of mental health stigma in online communication, consisting of naturally occurring online news and social media text annotated with a fine-grained taxonomy of stigma across multiple mental health conditions. Our annotation framework comprises a binary stigma-detection task and a multi-level taxonomy covering (i) stigma mode, (ii) domain, and (iii) specific components of certain forms of stigma. We apply this framework to texts mentioning six mental health conditions and evaluate large language models alongside stigma-related classifiers for detecting sentiment, toxicity, and hate speech. Results show that mental health stigma is not well captured by models trained to detect these neighboring constructs, and that LLMs often overpredict stigma unless given explicit operational rules - mirroring the importance of decision rules in human annotation

Expertise in vertical domains requires more than exposing LLMs to vast corpora; it demands the systematic learning of complex logic. Standard fine-tuning often treats data as an unstructured collection, failing to capture the dependencies needed for high-level reasoning. We propose Domain Knowledge Annealing (DKA), a curriculum learning paradigm designed to improve domain proficiency. Inspired by physical annealing, DKA organizes training from local to global: it starts by injecting discrete concepts via entity-centric samples and progressively advances to synthesizing complex, inter-document relationships. This progression allows the model to integrate isolated facts into a coherent knowledge system. Experiments on Linguistics and Law benchmarks show that DKA outperforms standard strategies, achieving state-of-the-art results among comparably sized models in domain-specific reasoning.

Hidden-state probes have been used to predict factual errors in language-model outputs. We study whether the same representations also distinguish four types of prompt-model relationship: known, unknown, ambiguous, and misleading (built on a false premise). On a controlled 200-prompt dataset constructed to balance six surface features, all six pairwise type contrasts are linearly recoverable from hidden states at the position where generation begins, at AUROC ≥ 0.986 on three open-weight instruction-tuned models spanning two families and a fourfold parameter range. The probe exceeds surface-feature, TF-IDF, and frozen sentence-embedding baselines on every pair. Under group-aware evaluation on paraphrased prompts with group-level shuffled controls, probe AUROC remains at least 0.949 on every pair while lexical baselines degrade. These findings frame hallucination as one downstream consequence of a four-type representational structure that also captures ambiguity and false premises.

Language models often appear to become insensitive to how text is tokenized. Models trained with different tokenizers can reach similar language-modeling loss, and prior work shows that they recover word-level information from the raw subword segmentations (a phenomenon referred to as "detokenization"). This raises a natural question: once that information has been recovered, do the resulting hidden states become independent of the tokenizer that produced them? We study this using *tokenizer twins*, pairs of models matched in architecture, training data, and random seed but trained with different tokenizers. We learn small translators between their hidden states at positions where the two tokenizers agree, and then test the same mapping where their tokenization paths differ. Transfer works well at agreement positions, adding less than one nat of loss, but breaks sharply at disagreement positions, where the loss increases by more than seven nats. This gap appears from the first hidden layer and persists through the network. At the same time, word-identity probes can still recover much of the underlying content from these representations. In other words, language models can abstract beyond their individual subword tokens while still carrying a lasting imprint of how the text was tokenized.

Aggregate refusal scores on mixed safe and unsafe prompts can obscure local boundary failures and label-correlated metadata shortcuts. We present PAIRSAFE, Proxy-Resistant Paired Safety Evaluation, a score-agnostic diagnostic protocol built from 150 deterministic, matched XSTest-derived safe/unsafe pairs. PAIRSAFE distinguishes strict pair locks from audited residual fields, then asks whether nuisance proxies collapse, unsafe twins rank above safe twins, and one threshold yields pair-consistent decisions; generated behavior is evaluated separately. Strict source, normalized-type, and pair-ID controls collapse to PSA 0.50 and PCDA 0.00, as required by construction. Residual metadata remain informative: a leakage-safe combined probe obtains nested Cal-PCDA 0.397, yet residualizing model scores against measured metadata leaves substantial paired signal. Direct-logit margins for Qwen2.5-3B-Instruct and SmolLM2-1.7B-Instruct reach PSA/Cal-PCDA of 0.987/0.880 and 0.973/0.760, respectively. Qwen’s secondary generation check is lower again (PCGA 0.700). These disagreements show why local ranking, shared-threshold decisions, and generated responses should not be collapsed into one score. PAIRSAFE is a bounded diagnostic protocol, not a new classifier or deployment certificate.

Large Language Models (LLMs) show impressive reasoning capabilities, yet their understanding of culturally specific social dynamics remains limited. We present KaiToM, a Theory of Mind (ToM) benchmark for Thai cultural contexts, focusing on Kreng-Jai, a uniquely Thai concept of social reluctance and consideration. Extending TactfulToM, we construct 882 conversations (39.2K questions) spanning three Kreng-Jai dimensions: White Lies, Avoidance of Confrontation, and Fear-Based Respect. To disentangle language proficiency from cultural knowledge, the dataset includes Thai scenarios at varying cultural depths, general English scenarios, and English translations of Thai content. We evaluate Thai-specialized (Typhoonv2.1, SEA-LION-v3), open-weight (Gemma-3, DeepSeek-V3.2, Kimi-K2, Qwen), and proprietary models (GPT-4o-mini, GPT-5.1, Gemini-3-Flash). Models achieve only moderate performance, struggle with Fear-Based Respect, and cannot reliably identify specific Kreng-Jai types, a failure shared by Thai-specialized models. Human evaluation shows that native speakers significantly outperform models on culturally loaded tasks. Performance also degrades in deep relative to surface-level cultural contexts, and generalist models often perform better on English translations than on the original Thai, suggesting that models capture surface cues but lack deeper cultural alignment. KaiToM supports the development of culturally aware systems for low-resource languages. Code is available at https://github.com/Augustus2011/KAI-ToM.

Humor judgment is subjective and often depends on ambiguity, expectation violation, and contextual reinterpretation, so the reliability of an LLM judge cannot be assessed from final detection performance alone. Existing LLM-based humor research and LLM-as-a-Judge studies largely focus on final performance, generation quality, or agreement with human evaluation, leaving prompt sensitivity and rebuttal robustness in subjective judgment tasks relatively underexplored. This paper analyzes how prompt design, structured interpretation, and user rebuttals shape LLM-based humor judgment. In zero-shot sentence-only judgments, models with similar F1 scores show substantially different FPR–FNR values, shifting between over-accepting and under-predicting humor depending on prompt formulation. Few-shot examples partially reduce these directional fluctuations, but their effects remain model-dependent. We further propose the Incongruity-Resolution Judgment framework, which compares a humor-supporting interpretation with a counter interpretation before producing a final judgment. Rather than uniformly improving performance, this framework serves as a diagnostic structure for examining how competing interpretations reshape model judgment behavior. Finally, explicit user rebuttals show that high initial performance does not guarantee robustness, highlighting the need to evaluate judgment persistence, directional error patterns, and interaction-level judgment trajectories.

Word-in-Context (WiC) remains challenging for language models, despite recent progress on lexical-semantic tasks. We hypothesise that this difficulty arises not only from comparing two contextual uses of a word, but also from the absence of an explicit sense inventory that specifies the relevant level of semantic granularity. We evaluate open LLMs on WiC and traditional Word Sense Disambiguation (WSD) under similar settings. We find that providing candidate senses, similar to what is done in traditional WSD, improves WiC performance in all settings. In general, explicit sense information helps models make more consistent and targeted judgements. Human evaluation further shows that many apparent WiC errors reflect label ambiguity or mismatches between model and annotator sense boundaries rather than simple failures of lexical understanding. In particular, results show that LLMs overthink the sense distinction often leading to errors based on overly fine-grained distinctions.

Brain tumor MRI diagnosis requires models not only to identify lesions but also to perform fine-grained medical reasoning based on imaging modality, tumor location, and key imaging findings. However, existing medical multimodal datasets often lack structured diagnostic semantics, and closed-ended evaluations may overestimate model ability due to option cues or elimination shortcuts. To address this issue, we introduce MM-NeuroOnco, a large-scale multimodal instruction dataset and benchmark for brain tumor MRI diagnostic reasoning, containing 24,726 MRI slices from 20 public data sources and approximately 200,000 multimodal instructions grounded in imaging evidence. We develop a multi-model collaborative pipeline for semantic completion and quality control, which supplements original tumor labels and segmentation masks with key diagnostic attributes, and further construct MM-NeuroOnco-Bench with a no-LLM-inference policy for ground-truth labels. The benchmark adds a “None of the above” option to closed-ended questions to reduce evaluation bias caused by option shortcuts. Experiments show that existing general-purpose and medical-specialist multimodal large models remain limited on this task, with the strongest baseline achieving only 41.88% accuracy on diagnosis-related questions. Fine-tuning on MM-NeuroOnco yields NeuroOnco-GPT, achieving a 27% absolute accuracy improvement and demonstrating the effectiveness of our dataset in enhancing fine-grained multimodal medical reasoning. Code and dataset documentation are available at https://github.com/gfnnnb/MM-NeuroOnco.

Large language models have made substantial progress in mathematical reasoning. However, benchmark development for multilingual evaluation has lagged behind English in both difficulty and recency. Recently, GSM-Symbolic showed a strong evidence of high variance when models are evaluated on different instantiations of the same question; however, the evaluation was conducted only in English. In this paper, we introduce MGSM-Pro, an extension of MGSM dataset with GSM-Symbolic approach. Our dataset provides five instantiations per MGSM question by varying names, digits and irrelevant context. Evaluations across nine languages reveal that many low-resource languages suffer large performance drops when tested on digit instantiations different from those in the original test set. We further find that models robustness in HRL setting do not necessarily translate to LRL. Moreover, proprietary models, such as Gemini 2.5 Flash and GPT-4.1 are less robust to digit, whereas Gemini 3.0 Pro is more robust. Among open models, GPT-OSS 120B and DeepSeek v3 show stronger robustness. Based on these findings, we recommend evaluating each problem using at least five digit-varying instantiations to obtain a more robust and realistic assessment of math reasoning.

Instruction-following benchmarks remain predominantly English-centric, leaving a critical evaluation gap for the hundreds of millions of Indic language speakers. We introduce IndicIFEval, a benchmark evaluating constrained generation of LLMs across 14 Indic languages using automatically verifiable, rule-based instructions. It comprises around 800 human-verified examples per language spread across two complementary subsets: IndicIFEval-Trans, translated prompts from IFEval carefully localized for Indic contexts, and IndicIFEval-Ground, synthetically generated instructions grounded in native Indic content. We conduct a comprehensive evaluation of major open-weight and proprietary models spanning both reasoning and non-reasoning models. While models maintain strong adherence to formatting constraints, they struggle significantly with lexical and cross-lingual tasks – and despite progress in high-resource languages, instruction-following across the broader Indic family lags significantly behind English. We release IndicIFEval and its evaluation scripts to support progress on multilingual constrained generation.

Recent studies challenge the reliability of cross-modal attention mechanisms as an indicator for visual token pruning in Large Vision-Language Models (LVLMs). In this work, we demonstrate that such negative conclusions can be substantially influenced by an overlooked phenomenon termed task-bound coupling, wherein cross-modal attention patterns become entangled with task-specific prompts. Through controlled configurations that decouple visual inputs from downstream queries, we observe that attention distributions can be relatively consistent across the evaluated settings. Concurrently, we identify an empirical scaling phenomenon which we term Attention Fidelity Decay, where the downstream effectiveness of attention maps exhibits semantic dilution in higher-capacity models or advanced architectures. Motivated by these insights, we introduce TAP-LLaVA, a simple training-free framework that leverages localized task-agnostic attention for efficient, reusable visual token pruning across multi-turn sessions. Our approach achieves competitive downstream performance while substantially lowering computational overhead. This suggests that attention can provide a reliable structural prior at moderate model scales when isolated from textual confounders.

Humor generation remains a challenging problem for language models due to its subjective, creative, and culturally sensitive nature. While large language models (LLMs) demonstrate strong generative capabilities, their deployment is often constrained by computational cost. We propose a lightweight, corrective feedback-driven framework in which a compact generator (GPT-2, 140M parameters) produces punchlines under the guidance of an LLM acting as a critic. Unlike traditional knowledge distillation, our method leverages the LLM as a feedback provider rather than a teacher to be mimicked, enabling the student model to iteratively refine its outputs through corrective supervision. We systematically evaluate multiple LLMs (LLaMA3-8B, Qwen3-8B and Mistral-7B) as guides under different strategies, including frozen and learnable configurations, to study their impact on humor quality and diversity. To support responsible humor generation, we curate a high-quality setup–punchline dataset from Reddit and explicitly filter out harmful content. Experimental results show that corrective feedback-guided training substantially improves humor quality, fluency, and diversity over standard fine-tuning and knowledge distillation baselines, while preserving the efficiency of compact models. We observe that, among the evaluated guides, Mistral-7B consistently provides particularly strong guidance on the curated joke dataset. Our code and resources are available at: https://github.com/Ramakrishna176420/Corrective-Feedback-Driven-Humor-Generation-for-Tiny-Language-Models.

Chain-of-Thought (CoT) prompting has significantly enhanced the reasoning capabilities of Large Language Models (LLMs). However, CoT can also produce fluent but logically flawed reasoning chains that make incorrect answers appear more reliable. In this work, we introduce REMIND (Reliability Enhancement via Metacognitive INtrospection and Debiasing), a framework that enables LLMs to learn from their own failures via episodic memory. Unlike standard error-correction methods that rely on external feedback or extensive fine-tuning, REMIND leverages a self-supervised introspection cycle. We first isolate epistemic failure—instances where CoT decreases model confidence despite yielding incorrect results. The model then performs structured introspection to diagnose the root cause and synthesizes a reusable natural language lesson. These lessons are stored in a vector database and retrieved during future inference steps to correct similar future errors. Experiments on eight diverse reasoning benchmarks spanning mathematical, scientific, multi-hop, truthfulness, and commonsense reasoning demonstrate that REMIND significantly improves reliability by effectively converting past CoT failures into actionable insights for unseen problems. Codes are available in https://github.com/newcodevelop/REMIND.

Reinforcement Learning with Verifiable Rewards (RLVR) has been shown effective in enhancing the visual reflection and reasoning capabilities of Large Multimodal Models (LMMs). However, existing datasets are predominantly derived from either small-scale manual construction or recombination of prior resources, which limits data diversity and coverage, thereby constraining further gains in model performance. To this end, we introduce DeepVision-103K, a comprehensive dataset for RLVR training that covers diverse K12 mathematical topics, extensive knowledge points, and rich visual elements. Models trained on DeepVision achieve strong performance on multimodal mathematical benchmarks, and generalize effectively to general multimodal reasoning tasks. Further analysis reveals enhanced visual perception, reflection and reasoning capabilities in trained models, validating DeepVision’s effectiveness for advancing multimodal reasoning.

Predicting next surgical phase, viewed as a form of surgical procedural intent, with high accuracy is essential for robotic real-time surgical systems. Previous approaches struggle with short-term dynamics, occlusions, and phase persistence; end-to-end large vision-language models (LVLMs) can reason from visual inputs but often fail to exploit structured temporal observations. Thus, we introduce SurgAgents, a multi-agent framework for embodied surgical intent prediction. SurgAgents decomposes prediction into specialized agents: (1) a ResNet-based single-frame classifier agent extracts per-frame phase information; (2) a YOLO-based instrument specialist agent extracts per-frame instrument sets; (3) a history descriptor agent composes these facts into frame-level descriptions; (4) a history summarizer agent compresses long histories into a compact temporal narrative; (5) a phase predictor agent assembles the narrative and the current frame into a formatted LVLM prompt; (6) a retrieval agent with a confidence-gated Retrieval-augmented generation (RAG) module selectively retrieves similar case patterns when prediction confidence is low. On Cataract-1K, SurgAgents attains 46.54% macro-F1 with GPT-4o, outperforming the baseline, with systematic ablation studies showing that each agent materially contributes to performance. Latency measurements demonstrate feasibility for robotic real-time surgical support during surgery procedures.

Large language models (LLMs) are increasingly used as role-playing dialogue agents in long-horizon applications such as tutoring, companionship, and narrative interaction, where users may expect stable persona consistency as well as plausible stage-conditioned change. Existing evaluations emphasize static persona fidelity, temporal factuality, or isolated social skills, but rarely test whether role-playing models produce coherent developmental differentiation across ordered stages. We introduce GROWTHEVAL, a psychology-inspired diagnostic framework for assessing stage-conditioned developmental realism in single-turn, stage-isolated role-playing responses. Using stage-conditioned scenarios and multi-axis scoring, GrowthEval evaluates cross-stage patterns in developmental appropriateness, role-play coherence, and canon fidelity across five socio-emotional dimensions. Across the evaluated LLMs, we observe a model-tiered empirical pattern: higher-capacity models respond more consistently to stage conditioning but can over-mature at early stages, whereas lower-capacity models exhibit flatter and weakly differentiated profiles. Reliability and robustness analyses provide evidence for the stability of these findings. Overall, the evaluated LLMs only partially realize coherent, stage-appropriate developmental change. GROWTHEVAL provides an extensible diagnostic framework for analyzing stage-conditioned behavioral differentiation and associated failure modes in single-turn role-playing responses.

The growth of context window lengths in Large Language Models (LLMs) significantly enhances their long-context capabilities but incurs prohibitive memory costs due to the Key-Value (KV) cache. Although low-rank compression of KV cache is a promising remedy, existing methods face a dilemma: offline approaches depend on external calibration data, whereas online approaches incur substantial compute for full-prompt decomposition and reconstruction. In this paper, we propose **S⁴R**, which builds low-rank subspaces from selectively sampled tokens and computes attention over a sparsely reconstructed KV representation. S⁴R uses prompt-aware initialization to build initial key/value bases from a representative prompt subset, trading off calibration-data dependence against prefilling cost. Because fully reconstructing the cache at every decoding step is prohibitively expensive and hurts throughput, we further adopt sparse reconstruction to retain only informative positions during decoding. Extensive experiments on LongBench and RULER with Llama and Qwen model families show that S⁴R achieves up to 5x KV compression with near full-cache accuracy, combining the efficiency of fixed compression with the adaptability of prompt-dependent methods. Our code is publicly available at https://github.com/Draconis98/S4R

Large language models are used by people with very different expertise levels, but existing work studies what users ask more than who is asking. We introduce QUILT (Query User Interaction Linguistic Taxonomy) , a two-dimensional framework that jointly characterizes LLM conversations by task type and user expertise level. Applying QUILT to 3,000 WildChat dialogues, we construct the first large-scale corpus linking expertise level to task-level interaction patterns in open-ended LLM conversations. Corpus analysis reveals clear expertise-dependent asymmetries: debugging queries are strongly associated with expert users, novice users rarely engage in reasoning-intensive tasks, and expert queries are substantially longer across all task types. These patterns are statistically significant and recoverable by automatic classifiers, indicating that expertise differences are systematically reflected in conversational language. Together, QUILT and the QUILT Corpus provide an empirical foundation for studying user heterogeneity in LLM interaction and for developing expertise-aware response strategies.

Evaluating retrieval-augmented generation (RAG) on custom knowledge bases requires task-specific benchmarks of multi-chunk evidence synthesis, yet most existing generation pipelines apply quality control only after the full question set has been solved. We present VERGE, which moves quality control inside the generation loop: a verification agent targets, per item and without an external solver pass, an operational multi-chunk criterion and a QA-integrity criterion, refining items that fail and discarding those it cannot repair. From 11,052 candidates across six domains, VERGE yields 2,854 verified multiple-choice items targeting aggregation- and comparison-style synthesis over topically coherent evidence. We contribute a method-agnostic validation protocol — exhaustive strict-subset ablation, chunk-removal ablation, format-unconfounded open-ended re-evaluation, and inter-chunk-similarity stratification — and use it to localise where the multi-chunk dependence holds. On a stratified sample (n = 250), 69.8% of items not answerable closed-book cannot be answered from any strict subset of their supporting chunks (strict-necessity violation rate 15.2%). A human annotation study finds the verifier reliable (Cohen’s 𝜅 = 0.903) and VERGE items higher in QA integrity and distractor quality than the closest prior method (p < 0.001).

A language model can fail a syntactic test in two distinct ways: by not encoding the relevant structure, or by encoding it but failing to use it at the output. Behavioral evaluation alone cannot tell these apart. We propose a three-level evaluation framework (behavioral deployment, LM-head readout, and probe recoverability) measured on the same items under the same binary decision. Using a compact trilingual (English, Chinese, German) control-dependency benchmark, we find that probe recoverability exceeds or equals LM-head readout, which in turn exceeds or equals behavioral deployment, across seven models and all three languages in the aggregate. The recoverability surplus is never negative across all 14 (model, task) conditions. The disconnect concentrates in subject-control, where a nearest-noun heuristic gives the wrong answer. The single largest gap (0.653) appears on Qwen3-0.6B Instruct in question answering. The gap persists at Qwen3-14B Instruct. Instruction tuning degrades deployment more than encoding in percentage terms. We rule out option-position bias, late-layer erasure, output-formatting artifacts, and probe-training variance. The pattern is consistent with decoding that favors surface shortcuts, and the behavior-probe gap measures the strength of that preference. Activation patching shows the gap is layer-localized. Under instruction tuning, the LM-head-decoded layer shifts approximately ten layers later than the probe-decoded layer. These findings argue that behavioral evaluation understates what models encode, while probing alone overstates what they deploy.

We audit whether benchmarks for LLM-based recovery of scientific assumptions measure corpus-grounded extraction or pretraining memorization. Using author-defined targets for 10 historical AI breakthroughs (frozen 4-case protocol plus 6 post-hoc stress cases) and a "field-wide paradigmatic assumption" prompt, GPT-4o’s top-10 extractions cover the target in 6/10 cases under a generous any-of-top-K criterion—an upper bound that a 10-rater study corroborates for only ∼0.19 of sampled top-ranked pairs (human–rule 𝜅=0.70, 𝛼=0.634). Controlled stress tests structure the diagnosis: a no-corpus control (field name only) still scores 4/4 on the primary shifts (memorization dominates), while a wrong-corpus control collapses to 0/12 across all 4×3 cross-pairings (corpus can be causally read); a five-LLM canonical sweep (GPT-4o 4/4, Qwen 2/4, Claude 1/4, Llama-3.1 1/4, Mistral 0/4) shows the gap is model-specific rather than closed-vs-open, with format compliance a major but not fully isolable contributor; a Min-K% Prob cross-check actually mis-ranks synthetic non-members as more contaminated than real pre-shift papers, motivating the behavioral framing; and synthetic/niche controls are mixed (causal in 1 of 3 synthetic scenarios, one shows corpus harm). We release an audit kit — five factorable controls (no-corpus, wrong-corpus, pair-level precision, ≥2-LLM, multi-seed; ∼$15) — as a reusable benchmark-validity probe, with all artifacts released.

Multimodal sarcasm detection aims to identify sarcasm in image-text posts by capturing how visual content changes the interpretation of short informal text. Existing methods have advanced the task through multimodal fusion, caption-based cues, and fact/sentiment incongruity modeling, but their evidence use is often controlled inside latent feature channels or reduced discrepancy scores. This makes it difficult to decide which visual information should be selectively weighted for a particular post. We propose CAFE, a caption-anchored feature enrichment and evidence routing framework that decides how to use visual evidence for each sample under ambiguity. CAFE enriches each image with two generated descriptions, one for visible facts and one for the image-text relation, then estimates how useful each description is for the current post and fuses caption-calibrated contrast evidence with direct cross-modal evidence for prediction. Experiments show that CAFE achieves state-of-the-art performance on the primary MMSD2.0 benchmark with 87.81+/-0.33% F1 (mean over 5 seeds). Component ablations on the original MMSD provide an additional evaluation.

Sparse autoencoder (SAE) features are increasingly used to interpret language models, with auto-generated natural-language labels serving as the primary interface for understanding what each feature represents. We ask whether these labels generalize: does a feature labeled for a concept actually track that concept across languages and scripts? Using Serbian digraphia as a controlled testbed—the same language written in both Latin and Cyrillic via deterministic transliteration—we first find that SAE feature sets activated by the same content in different languages, scripts, and wordings share substantial overlap (mean Jaccard 0.39 vs. 0.13 random baseline, peaking at 0.57), suggesting genuine cross-lingual semantic features. We then test whether auto-interpretation labels keep pace. They often do not: features whose labels describe semantic content miss the same meaning in Serbian up to 4× more often than within English, and miss Serbian Cyrillic more than Serbian Latin—two scripts that are deterministic transliterations of each other. The gap grows with network depth, yet the labels give no indication that they fail. These results suggest that auto-interpretation labels reflect a feature’s behavior on the languages and scripts a model has seen most in training, rather than the concept itself.

Using large language models (LLMs) to simulate diverse human populations promises to transform computational social science. However, the accuracy of these simulations remains unclear because current evaluations prioritize predicting an average response over capturing the rich diversity of opinions within real human groups. Here, we develop a diagnostic framework measuring both accuracy and viewpoint retention across 10,000 respondent-question pairs from the World Values Survey, spanning twelve countries and six continents. By evaluating nine LLMs across four alignment paradigms (zero-shot, SFT, DPO, GRPO) against ground-truth human responses, we identify a failure mode we term consensus collapse. While training alignment improves point accuracy, we find that it compresses the range of outputs toward a single demographic stereotype. The highest-performing model (Tulu 3 70B DPO, 57.9% accuracy) retains only 50% of real human variance (σr = 0.50). We demonstrate this collapse is within the models’ underlying parameters rather than decoding procedures; temperature scaling up to T = 1.0 alters fewer than 3.7% of predictions. Furthermore, this diversity loss is inequitable, erasing significantly more variation for Global South populations like Nigeria (σr = 0.11) and Ethiopia (σr = 0.16) compared to Western democracies (σr = 0.70–0.87), with accuracy-rewarded GRPO unable to reverse this effect. While mixing an aligned model with an unaligned prior partially mitigates the collapse (reducing W1 from 0.55 to 0.52), substantial gaps remain (Nigeria recovering from σr = 0.14 to 0.47). Ultimately, we find that modern AI alignment methods trade distributional diversity for point accuracy, challenging existing assumptions for using LLMs as substitutes for human populations. Yet, by exposing the mechanics of this collapse, our work provides insight for future methods designed to value and preserve diverse human viewpoints.

We argue, with systematic empirical evidence, that a large language model’s political ideology is not a fixed point, but a *conditional distribution* ℙ(position∣context) over a real political space. We evaluate nine current LLMs using a unified measurement framework anchored by VAA–CHES projection models, which map responses onto three validated dimensions (*lrgen*, *lrecon*, *galtan*) across six contextual axes. Our findings reveal high sensitivity to context: persuasive framing and under-represented languages displace coordinates by up to 0.57 and 0.52 units, respectively, while chain-of-thought reasoning often amplifies rather than dampens paraphrase instability. Despite this local plasticity, the model cohort occupies a remarkably narrow Overton envelope overall, occupying roughly one-third the spread of major European parties. Supported by a multi-trait multi-method (MTMM) analysis, we conclude that a single point cannot summarize LLM political behavior; it must be characterized as a shape. Our code and data are publicly available.

Prompt design affects the moral judgments and refusal behavior produced by large language models (LLMs), yet controlled comparisons remain fragmented across datasets and models. We introduce ProMoral-Bench, a benchmark-style comparison of 10 prompting paradigms across four LLM families. Using ETHICS, Scruples, WildJailbreak, and our 200-pair ETHICS-Contrast set, we report task-level metrics together with Moral Competence Score (MCS), jailbreak/over-refusal Safety Robustness Score (SRS), and Unified Moral Safety Score (UMSS). UMSS is a study-specific relative summary of MCS and SRS, not an absolute measure of model safety or moral grounding. In this fixed-template evaluation, compact exemplar-guided scaffolds attain stronger competence-robustness trade-offs at lower token cost than the particular multi-stage variants tested, although multi-stage prompting occasionally improves jailbreak refusal. The few-shot demonstrations are non-overlapping but drawn from the evaluation distributions, so their advantage should not be read as out-of-distribution robustness. Code and data are available at https://github.com/rohanthomas334-sketch/ProMoral_Bench.

Large Language Models (LLMs) exhibit remarkable capabilities but remain susceptible to adversarial attacks aimed at eliciting harmful responses. Current defense mechanisms, while partially effective, often compromise the models’ performance on benign tasks. To address this issue, we propose a novel two-stage framework to enhance LLM robustness against adversarial attacks. Initially, we train a universal adversarial prompt designed to align the hidden representations of harmful inputs with benign ones, effectively emulating adversarial attack patterns. Subsequently, we utilize this soft prompt to augment training data, strengthening the model’s capability to reject harmful content. We conduct extensive evaluations to empirically demonstrate that our method outperforms existing methods such as Circuit Breaker (CB) and Deliberative Alignment SFT (DSFT), achieving average defense rate of 96.4% on llama3-8b-instruct model, compared to CB (90.6%) and DSFT (87.9%).

Large Language Models (LLMs) have shown potential in simulating human behaviors and performing theory-of-mind (ToM) reasoning, crucial for complex social interactions. We investigate ToM reasoning’s role in aligning agentic behaviors with human norms in negotiation tasks, using the ultimatum game as our referenced task. We initialized LLM agents with different prosocial beliefs (Greedy, Fair, Selfless) and reasoning methods (chain of thought and ToM reasoning of varying levels), examining their decision-making processes and outcomes across multiple LLMs, including reasoning models like o3-mini and DeepSeek-R1 Distilled Qwen 32B. We perform 2,700 simulations to show that ToM reasoning enhances behavioral alignment with human norms, decision-making consistency, and negotiation outcomes. Consistent with prior findings, reasoning LLMs exhibit limited capability compared to ToM-enhanced LLMs, with different game roles benefiting from different ToM orders.Fair proposers and responders accepting offers were the most consistent with their strategic reasonings, whereas all agents showed strong consistencies with human beliefs when rejecting offers, except when the offer was fair. Human verification further revealed that Llama 3.3 70B produces reasoning most consistent with its actions and beliefs. Our findings advance understanding of ToM’s role in human-AI interaction and cooperative decision-making. The code used for our experiments can be found at https://anonymous.4open.science/r/UltimatumToM.

Large language models rely on prompt context to perform in-context learning, follow formats, and adapt to local evidence. This ability, however, also raises a robustness concern: models may reuse irrelevant tokens from earlier context when answering a later query. Prior work identifies this behavior as contextual entrainment and finds attention heads whose ablation reduces it, arguing that these heads are largely task- or relation-specific. But are the mechanisms behind contextual entrainment truly isolated by relation, or do relation-conditioned searches recover reusable components shared across relations? Using factual-distractor relation-completion prompts across 19 lexical-semantic subtypes, we identify attention heads whose ablation reduces entrainment and test how the resulting head sets transfer recovery ability across exact relations, semantic families, and mixed relation distributions. Same-family transfer exceeds different-family transfer, family residuals weaken once globally shared heads are removed, and subtype-specific searches repeatedly rediscover many global heads. Clean-completion and context-use probes further suggest that this shared intervention can reduce harmful copying while preserving ordinary relation completion and simple in-context abilities. Our findings provide a more precise perspective on contextual entrainment and its mitigation: harmful context following is not best addressed by searching for isolated relation-specific circuits, but by using heterogeneous attack prompts to reveal reusable intervention targets whose effects are shaped by semantic relation structure.

Text simplification aims to modify documents and sentences to make them easier to read and understand while preserving their meanings. Different applications require distinct simplification policies, e.g., lexical-oriented simplification and overall rewriting while trading off detail for simplicity. However, achieving such policy-driven control remains an open challenge. In this work, we introduce a simple yet powerful approach that leverages Large Language Model-as-a-Judge (LLM-Judge) to automatically construct policy-aligned training data, removing the need for costly human annotation or parallel corpora. Our method enables the construction of simplification models that adapt to diverse simplification policies at both the sentence and document levels. Remarkably, small-scale open-source LLMs match or surpass GPT-5, as verified by both automatic metrics and edit analysis. The consistent improvements across model families and sizes demonstrate the robustness of our approach.

Pretrained text embeddings are increasingly used as representational maps, yet high category separability does not imply that their geometry recovers expert-defined structure. We study this problem in mental-health-related language, where symptom relations provide an external reference and online communities introduce strong domain, affective, stylistic, and discourse confounds. Using 28 Reddit communities, we compare pretrained and supervised fine-tuned Qwen3 embedding spaces at two scales (0.6B and 4B). We construct category prototypes, evaluate their representational dissimilarity matrices against an expert symptom matrix with representational similarity analysis, and complement this global test with prototype-based typicality and multi-baseline confound controls. Pretrained embeddings show measurable alignment with expert structure within the mental-health subset; fine-tuning strengthens this alignment most at the finest category level; and larger scale improves both zero-shot alignment and supervision-induced gains. Residual alignment remains substantial after controlling for VAD, LIWC, lexical style, and topic-distribution structure. These results suggest that LLM embeddings can recover expert-relevant category geometry, but this recovery is level-dependent and should be tested against explicit confounds rather than inferred from classification alone.

Large Vision-Language Models (LVLMs) achieve strong multimodal reasoning capabilities, yet are still prone to object hallucination - often generating content that is not grounded by the visual input. Existing attention-based training-free mitigating methods rely on visual attention heads identified by high attention weights or sensitivity to visual token perturbations. However, we show that these identified visual attention heads are not necessarily responsible for visual grounding, and using this as guidance is suboptimal. Motivated by this observation, we propose a novel training-free decoding method that identifies visually sensitive heads by counterfactual region masking. We find that these heads provide a clear signal when visual grounding is required and can therefore be used to mitigate hallucination by guiding the suppression of hallucination heads. Across multiple LVLMs and benchmark image captioning and VQA datasets, our method achieves best or near-best performance in reducing hallucination while preserving generation quality.

Missing and misleading evidence are not the same problem in retrieval-augmented generation (RAG), but prompt-based abstention treats them alike. Small frozen models abstain when context is absent, not when it is misleading. We introduce GRAB-RAG (Graded Abstention Benchmark for Retrieval-Augmented Generation), a paired benchmark that tests the same questions across four context conditions (supportive, degraded, missing, and misleading) in Natural Questions and HotpotQA. In the misleading condition, we edit a gold passage to support a wrong answer and place it among other retrieved passages. We test five abstention policies on three small frozen models (3.8B–8B) across two English QA benchmarks. Models abstain reliably when evidence is missing, but under explicit abstention prompting still answer 41.6% of misleading questions, with 63% of those answers echoing the planted wrong entity. Chain-of-thought provides little additional benefit. A generator-side conflict check cuts the rate to 13.3% but discards many correct answers, while a natural language inference (NLI) verifier recovers that coverage but fails when parametric memory and the misleading passage agree on the same wrong answer. Prompt-based abstention asks whether context is sufficient, not whether it is correct. The two-pass checks are diagnostic baselines, not deployable mitigations.

Adaptive learning systems commonly formulate learning path planning as Exercise-Centric (EC) recommendation, where the next step is inferred from item-level interaction logs. Evaluating goal-oriented guidance additionally requires explicit learner goals and curriculum-scale prerequisites: learners with similar exercise records may need different paths toward their targets. We therefore study Knowledge-Centric (KC) personalized learning path planning, where a planner must reason over learner profiles, mastery states, and prerequisite knowledge structures to decide which textbook, unit, and concept should be studied next. To support this setting, we introduce PersonaPath, a benchmark that pairs 2,000 fine-grained learner personas with a hierarchical knowledge graph of 347 textbooks, 1,751 units, and 4,092 concepts across 77 subjects. We evaluate representative LLMs on PersonaPath. Results show that even the strongest LLM reaches only a 29.5% final pass rate in Basic Education, and that the main bottleneck lies in adaptivity, where no model exceeds 44.7% in tailoring paths to individual learners.

The rapid advancement of Large Language Models (LLMs) has necessitated more robust evaluation methods that go beyond static benchmarks, which are increasingly prone to data saturation and leakage. In this paper, we propose a dynamic benchmarking framework for evaluating multilingual and multicultural capabilities through the social deduction game Spyfall. In our setup, models must engage in strategic dialogue to either identify a secret agent or avoid detection, utilizing culturally relevant locations or local foods. Our results show that our game-based rankings are consistent with Chatbot Arena rankings on the overlapping model. However, we find a significant performance gap in non-English contexts: models are generally less proficient when handling locally specific entities and often struggle with rule-following or strategic integrity in non-English languages. We demonstrate that this game-based approach provides a scalable, leakage-resistant, and culturally nuanced alternative to traditional NLP benchmarks. The dataset (game history) and source code can be accessed here.

In this work, we introduce BAREC-X, the first explanation-annotated Arabic readability dataset, comprising 1,793 sentences drawn from the Balanced Arabic Readability Evaluation Corpus (BAREC) and labeled by expert annotators with linguistic justifications aligned with the BAREC guidelines. Annotators reached an average inter-annotator agreement of 93.3% under the At-Least-1 metric (73.9% Exact Match at the Category+Reason level, and 81.9% at the Category level). We frame readability explanation as a gold-level-conditioned classification task: given a sentence and its gold readability level, a system must return the linguistic reasons that justify the label. We present the first systematic benchmark for this task on Arabic. We evaluate two contrasting paradigms: (i) eight LLMs under zero- and few-shot prompting, and (ii) a fully interpretable, training-free rule-based system grounded in 67 linguistic features spanning morphology, syntax, vocabulary, and syllabic structure. Gemini 3 and GPT-5.5 achieve the highest exact-match agreement on fine-grained reason prediction (55.8% and 56.4% Category+Reason), while the rule-based system achieves the strongest exact-match performance at the Category level (65.9%). Together, BAREC-X and our benchmark establish a transparent foundation for interpretable Arabic readability explanation.

Audio large language models (LLMs) enable unified speech understanding and generation, but adapting them to linguistically complex and dialect-rich settings such as Arabic-English remains challenging. We present a controlled study of multi-task instruction tuning for an Arabic-centric audio LLM across generative tasks including ASR and speech and text summarization, and discriminative tasks including dialect and emotion recognition, in a resource-constrained setting. To support end-to-end Arabic speech summarization, we introduce AraMega-SSum, a first speech summarization resource for training and benchmarking Arabic-centric Audio-LLMs. We compare four training strategies (i) Uniform Mixing (UM), (ii) Task-Progressive Curriculum (TPC), (iii) Aligner-Based Diverse Sampling (ADS) for training-time batch construction, and (iv) A two-stage TPC→ADS strategy. Our results show a clear task-dependent trade-off: TPC and uniform mixing favor generative tasks, while ADS is strongest for SER. The two-stage TPC→ADS strategy achieves the best DID performance across both backbones while preserving competitive generative quality. The adapted models also outperform zero-shot proprietary baselines, including Gemini-2.5-Pro, on discriminative tasks. We made AraMega-SSum and all experimental resources available to the community (https://huggingface.co/datasets/QCRI/AraMega-SSum).

Post-training quantization facilitates efficient LLM deployment, yet it disproportionately erodes reasoning and mathematical performance. To recover fidelity without costly fine-tuning, we identify a critical vulnerability in existing output-space methods: the tendency to overfit the limited calibration manifold, which necessitates structural regularization for unseen data. We propose **ROCOB** (**R**esilient **O**utput **C**ompensation with **O**ptimal **B**udgeting), a training-free framework based on *Dual-Manifold Alignment*. ROCOB couples a Principal Activation Component (PAC), which minimizes error on the calibration manifold, with an Orthogonal Geometric Component (OGC), which recovers residual weight-space structure without fitting a second calibration objective. To compare methods fairly under the same added-parameter budget, ROCOB uses Global Spectral Energy Maximization (GSEM) to allocate ranks according to recovered spectral energy and module cost. Across Qwen3 and Llama-3 models, quantizers, and reasoning benchmarks, ROCOB improves average post-quantization recovery in most evaluated model–quantizer settings while preserving a training-free construction pipeline.

Performing causal reasoning akin to humans is fundamental to achieving strong artificial intelligence. However, existing studies on causal reasoning and large language models (LLMs) often underestimate the role of human causal cognition. Human causal reasoning is far more than recognizing causes and effects. Humans often factor in many other considerations when establishing a causal link between two events, such as norm violations, intentionality, morality, counterfactual scenarios, etc. To simulate such reasoning, it is intuitive to identify cause-effect pairs using token causality formalisms while modeling the factors that influence human causal judgment using findings from cognitive science experiments. To support the evaluation and modeling of human-like causal reasoning in LLMs, we introduce HCR-Bench, a challenging benchmark with 1,093 instances and detailed human causal reasoning step annotations, and HCR-Reasoner, an approach that systematically integrates theories of token causality and cognitive science into LLMs for human-like causal reasoning. Results suggest that HCR-Reasoner consistently improves LLMs’ causal alignment with humans, and that theory-guided reasoning is highly effective.

With the emergence of large language models (LLMs), there is an expectation that they can effectively extract structured information from complex, real-world documents. However, most LLMs generate paragraph-style answers that are chaotic, disorganized, and untraceable. To bridge this gap, we introduce the Arranged and Organized Extraction Benchmark (AOE), a new bilingual benchmark featuring long-form documents (171k tokens on average) designed to systematically evaluate the ability of LLMs to comprehend fragmented information and reconstruct it into an organized table. Unlike conventional text-to-table tasks, which rely on fixed schema and narrow domains, AOE includes 12 carefully designed tasks across four diverse domains, requiring models to generate context-specific schema tailored to varied input queries. In our experiments, we evaluated 12 open-source and proprietary LLMs. Even the most advanced models struggle significantly.

Large language models (LLMs) have advanced argument mining (AM), yet most approaches treat component classification, relation identification, and explanation generation as separate tasks, limiting coherence and explainability. We propose the first explainable end-to-end AM framework grounded in Toulmin’s theory, jointly predicting argument components, argumentative relations, and structured explanations in a single generation step. This unified formulation encourages LLMs to produce internally consistent argument graphs and reasoning traces aligned with their predictions. Further, we introduce five explainability criteria tailored to AM and grounded in Toulmin’s theory: completeness, correctness, faithfulness, soundness, usefulness, and develop a unified evaluation protocol combining structural metrics, neural similarity, LLM-based judgment, human evaluation and qualitative analysis. Extensive evaluation on several LLMs shows that our approach achieves competitive performance while substantially improving explainability, generating Toulmin-based explanations that are complete, grounded and faithful, uncovering reasoning errors not captured by standard AM metrics, and being consistently preferred in usefulness evaluations.

Large language models (LLMs) have recently shown promise as planners across web, spatial, and embodied domains. However, existing agents often handle information seeking, spatial grounding, and action planning as separate capabilities, limiting their ability to solve long-horizon compositional tasks. To address these challenges, we present **WebWorld**, a web-simulated world model for long-horizon reasoning over web evidence, scene observations, and action plans. WebWorld maintains structured intermediate states across heterogeneous modalities and verifies them through schema-guided hierarchical planning. The framework introduces a Mixture-of-Transformers architecture that coordinates website, scene, and motion experts via multi-modal self-attention. To support training and evaluation, we develop WebWorld-CoT-55K, a large-scale dataset that integrates heterogeneous modalities into schema-guided reasoning traces, and **WebWorldBench**, a curated benchmark for evaluating long-horizon compositional tasks. Training combines cold-start supervised fine-tuning with offline reinforcement learning based on structural, semantic, and planning rewards; inference uses test-time scaling to aggregate multiple reasoning trajectories. Experiments on WebWorldBench show that WebWorld improves the overall score by up to 8.4% over strong LLM-agent baselines, with gains in outdoor reasoning, geolocation, and travel planning, while still leaving substantial room for progress toward human-level performance.

Recent diffusion and autoregressive video generators have substantially improved short-form synthesis, yet composing multiple independently generated clips into a coherent audiovisual story remains challenging. Character identities may drift across scenes, narrative events can become inconsistent, and dialogue or sound may not align with the evolving visual context. We present **DramaAgent**, a hierarchical, agentic, and model-agnostic framework for multi-scene long-horizon audiovisual story generation. Rather than modifying the underlying video backbone, DramaAgent introduces a stateful control layer that combines hierarchical story planning, persistent character conditioning, scene-wise generation, and reflection-guided targeted repair. The framework maintains reusable story and character states across scenes, diagnoses failures along identity, semantic, temporal, and audio-visual dimensions, and maps each diagnosed failure to a corresponding localized repair operation. Experiments on 50 narrative scripts across five video-generation backbones show consistent improvements in semantic alignment, character consistency, narrative fidelity, and scene-level audio-visual compatibility. Controlled same-backbone comparisons further show that these gains cannot be explained solely by stronger generators or additional sampling. Our results suggest that persistent state, failure diagnosis, and targeted repair provide a practical control paradigm for coherent multi-scene audiovisual generation.

Lexical evolution is universal across languages, yet the mechanisms driving neologism formation vary substantially from one language to another. In Chinese, this variation is particularly reflected in the productivity of compounding, where morphemes are combined through word-formation relations to create interpretable lexical forms. This morphological productivity implies that the Chinese lexicon can be viewed not only as a set of attested words, but also as a dynamically latent space with potential neologisms to be realized. However, this perspective still remains underexplored in computation, as current approaches are largely reactive, focusing on detecting attested words from corpora. This raises a question about word-formation itself: which morpheme combinations do the rules license beyond those that corpora already attest? Motivated by this, we propose a proactive creation paradigm for Chinese neologism research. We introduce a three-stage pipeline: KG-Based Candidate Generation, Morphological Discarding, and LLM-Based Filtering. We then probe the candidates with three complementary signals: expert judgment, model perplexity, and corpus attestation. The three signals agree: accepted candidates sit close to real words and clearly apart from rejected combinations. The pipeline yields 18,662 candidates with definitions, examples, and formation metadata, released as material for follow-up work, such as lexical gaps and lexical change. Overall, this study substantiates a proactive creation paradigm for Chinese neologism research, and uses it to characterize what Chinese word-formation rules license beyond what dictionaries have already recorded.

Reliable model fingerprints are essential for protecting large language models (LLMs) against unauthorized redistribution and commercial misuse. In black-box deployment, verification is hindered by defensive filtering of suspected fingerprint queries, as well as by downstream model modifications that may weaken embedded ownership evidence. These risks require fingerprints to be robust in both construction and injection. For construction, prior paradigms face an imperceptibility trade-off: natural-language fingerprints may be accidentally activated, whereas garbled fingerprints are statistically exposed and easier to filter. For injection, existing methods struggle to preserve persistent trigger–target behaviors under model modification. We propose an end-to-end injected fingerprinting framework to address these challenges. Code-mixing Fingerprints (CF) use lowest-perplexity code-mixing under a high-complexity constraint to mitigate this two-sided imperceptibility trade-off. Multi-Candidate Editing (MCEdit) constructs structurally redundant, margin-separated trigger–target mappings to enable graceful degradation under model modification. Extensive evaluations on imperceptibility, detectability, and harmlessness demonstrate robust ownership verification with negligible impact on utility.

Automated fact-checking is a crucial task that supports a responsible information ecosystem. While recent research has progressed from text-only to multimodal fact-checking, a prevailing assumption is that incorporating visual evidence universally improves verification accuracy. In this work, we challenge this assumption and show that the indiscriminate use of visual evidence can reduce accuracy. Building on this finding, we propose AMuFC, a modular fact-checking framework that employs two collaborative vision-language models with distinct roles to enable the adaptive use of visual evidence. Experimental results on three datasets, including WebFC, introduced in this study, demonstrate the effectiveness of adaptive visual evidence use in fact-checking.

On-policy distillation (OPD) for long-horizon language agents fails even when the teacher remains sequence-informative: in a failing Qwen3 configuration, the teacher still separates good from bad rollouts (sequence AUROC ≈ 0.75), yet the local token-gradient alignment 𝛼_diag decays from 0.71 to 0.09 and standard OPD recovers only 1.3% of the teacher–student gap. We restrict our claims to two coupled mechanisms, jointly termed CHOP (Counterfactual Hierarchical On-Policy distillation): (i) a per-state reliability gate routing between per-token reverse Kullback–Leibler matching and per-segment sliced-Wasserstein matching, and (ii) a counterfactual total-variation gate activating privileged information (PI) only when the PI changes the teacher’s distribution. The scope is explicit: CHOP targets low-reliability, long-horizon traces where token-local support matching plateaus; in short-horizon or high-reliability regimes Top-K local-support matching (LSM) is already sufficient and CHOP’s segment branch is rarely activated. CHOP recovers 31.7% of the anisotropic-Qwen3 gap (vs. 1.3%/14.5%/24.7% for OPD/Top-K LSM/Li et al. 2026); on SWE-bench Verified under a fixed OpenHands harness, CHOP-deployable prevents the OPD-induced regression (6.5%\!→\!13.2%, matched to off-the-shelf 11.8%, not a significant gain over off-the-shelf) and oracle PI reveals further headroom (14.2%, 95% CI [11.4,17.3]); on AIME 2024 at a 60K-token stress cap (a diagnostic horizon, not a deployment target), CHOP holds 44.9 vs. 42.8 for Top-K LSM (p=0.032), while standard OPD collapses to 28.4. Compute-matched, leakage, and PI-removed-teacher controls attribute the gain to supervision-object routing rather than extra compute or PI contamination.

Existing paradigms for black-box LLM evaluation by independent third parties share a structural weakness: the testing benchmark must reach the model being evaluated, enabling model developers to game future evaluations through logging or retraining. Ideally, rigorous evaluation would grant evaluators direct access to the model so testing can occur without exposing benchmark contents. However, proprietary black-box models cannot be released to external evaluators, creating a fundamental barrier to independent evaluation. To address this gap, we propose the Private Evaluation Protocol (PEP), a surrogate-based framework for independent evaluation of black-box models without sending the private benchmark to the model developer. PEP releases a small public slice of a held-out benchmark, queries the target model only on that slice, and trains a surrogate from the resulting responses, balancing fidelity to the target model against leakage of the private remainder over a tunable revelation budget. Across six benchmarks, composite leakage scores reliably predict behavioral exposure (r=0.788), and fine-tuned surrogates reach 0.85–0.90 agreement on template-heavy datasets while plateauing below 0.73 on reasoning-heavy datasets. The optimal revelation budget is dataset-dependent, and PEP estimates black-box performance with 5.88 percentage-point MAE, showing the feasibility of private surrogate-based evaluation for structured tasks while protecting the privacy of the held-out benchmark.

Large language models (LLMs) face significant challenges in sustaining long-term memory for agentic applications due to limited context windows. To address this limitation, many work has proposed diverse memory mechanisms to support long-term, multi-turn interactions, leveraging different approaches tailored to distinct memory storage objects, such as KV caches. In this survey, we present a unified taxonomy that organizes memory systems for long-context scenarios by decoupling memory abstractions from model-specific inference and training methods. We categorize LLM memory into three primary paradigms: natural language tokens, intermediate representations and parameters. For each paradigm, we organize existing methods by three management stages, including memory construction, update, and query, so that long-context memory mechanisms can be described in a consistent way across system designs, with their implementation choices and constraints made explicit. Finally, we outline key research directions for long-context memory system design.

Large Language Models (LLMs) increasingly operate in multi-turn interactions that require maintaining latent conversational states, not only explicit facts. We study affective state as an implicit state inferred from sparse contextual cues that remain relevant after topic shifts, and ask whether LLMs expose internal attention signals for routing such evidence at inference time. Prior attention interventions rely on labeled in-domain head profiles and externally specified target spans, limiting transfer across affective-evidence distributions. We propose Self-targeted Internal Attention Boosting (SIAB), which localizes sparse affective heads via causal localization (in detail activation patching) on synthetic minimal pairs and boosts, at the answer position, the content tokens these heads already attend to; both edited heads and target tokens are derived from the model’s own behavior, enabling the same head set to be reused across tasks.We evaluate on ECC and GoEmotions across three LLMs and multiple prompting settings. Across settings, SIAB improves prompt-matched baselines by up to +10.3 macro-F1 on ECC and +5.5 on GoEmotions. Combined with PASTA-style external supervision, it achieves up to +13.84 accuracy points and +12.51 macro-F1 points, while mechanistic analyses trace the gains to sparse, emotion-enriched heads and show that layer proximity affects additivity.

Large language models can be updated after training by modifying parameters, conditioning on edit demonstrations, or supplying retrieved evidence. Yet successful factual updates should do more than change one prompt: they should generalize to related questions, transfer across languages, suppress obsolete facts, and preserve unrelated knowledge. This is the problem of measuring cross-lingual ripple effects of factual updates. The existing evaluations on multilingual knowledge injection, however, focus on high-resource languages like English, and it is unknown whether the findings can transfer to Indic languages. In this paper, we present IndicRipple, a controlled evaluation of multilingual factual updating over 500 factual updates across five domains and five languages: English, Hindi, Marathi, Telugu, and Kannada. We compare parametric editing, in-context editing, and retrieval-based correction using ripple-oriented probes for logical generalization, compositionality, subject aliasing, relation specificity, and forgetfulness. Our results show that current factual update mechanisms show limited cross-lingual ripple effects. IKE performs best overall but reaches only 21.07% strict semantic-plus-language accuracy; RAG often recovers the edited fact but fails to answer in the requested language; and ROME/MEMIT are frequently language-compliant without semantically propagating the update. Strict accuracy is consistently higher for English evaluation than for the Indic-language evaluations. These findings show that direct update success overstates progress: multilingual factual updating must be evaluated as stable cross-lingual propagation, not isolated prompt-level overwrite.

Speaking-style captioning (SSC) aims to generate natural language descriptions of how speech is delivered, capturing paralinguistic attributes such as vocal timbre, prosody, and expressivity. Many downstream applications, including conversational AI agents, controllable speech generation, and large-scale audio annotation, require dimension-specific style captions that describe targeted aspects of speech (e.g., speaker traits, emotion, or delivery style), rather than a single undifferentiated description. However, existing work lacks a unified formulation for controllable speaking-style captioning. We address this gap by formulating SSC as an instruction-following audio-language task, where explicit instructions specify the speaking-style dimensions to be described. We introduce StyleInstructCaps, the first standardized benchmark for controllable speaking-style captioning in audio-language models. StyleInstructCaps provides a dataset and evaluation framework that measures metadata groundedness, hallucination, instruction-following ability, generalization to unseen instructions and audio datasets, and speaker-style consistency. Across nine state-of-the-art audio-language models, we find that strong instruction-following ability does not necessarily imply grounded speaking-style understanding, and that SSC-specific instruction-diverse training substantially improves groundedness, controllability, and speaker consistency.

Large Language Models (LLMs) have significantly advanced text generation capabilities, including tasks like summarization, often producing coherent and fluent outputs. Despite these advances, hallucinations remain a critical challenge. While prior studies primarily focus on detecting or mitigating hallucinations, little attention has been paid to where hallucinations occur within long generated outputs. In this work, we investigate where hallucinations occur in LLM-based long response generation, using long document summarization as a key case study. Focusing on the challenging setting of long context-aware long response generation, we find a consistent and concerning phenomenon: hallucinations tend to concentrate disproportionately in the latter parts of the generated long response. To understand this bias, we explore potential contributing factors related to the dynamics of attention and decoding over long sequences. Furthermore, we investigate methods to mitigate this positional hallucination, aiming to improve faithfulness specifically in the concluding segments of long outputs.

Transactional agents must act on underspecified requests. "Book me a flight from Chicago to San Francisco" omits budget, timing, stop tolerance, and airline loyalty, any of which can change the best option by hundreds of dollars. The challenge is identifying, without a user-specific prior, which unstated attribute preference is most likely to change the recommendation.

We introduce ELICITBENCH, a benchmark of 1,387 travel and housing scenarios pairing real search-result pages (SERPs) with transaction-grounded preference profiles. Each scenario partitions attributes into stated and hidden preferences. An ex-post analytical oracle assigns question-value labels based on the normalized reduction in market-conditioned recommendation regret. Agents are evaluated by task success, question count, and net utility under a super-linear patience cost.

ELICITBENCH reveals that plausible questions are often not decision-relevant: on Gemini 2.5 Flash and Claude Sonnet 4.5, unguided agents ask natural follow-up questions with little downstream value. SERP-aware prompting raises net utility from 0.073 to 0.306 and task success from 21.7% to 57.6% on Gemini, and net utility from 0.221 to 0.346 on Sonnet while asking fewer questions. The selection of items is a second bottleneck: with every preference stated, agents still pick the wrong item in about 11% of the scenarios.

We investigate how small open-weight language models make decisions in medical tasks and whether their failures also occur in other settings. We evaluated nine models on wearable fatigue and intensive-care tasks, where they chose among six recommendations, including continued monitoring. We then extended testing to a non-medical tool router and subsets of the Berkeley Function-Calling Leaderboard and JSONSchemaBench. We introduce the Minimum Viable Audit (MVA) to examine which actions models choose, whether their responses contain usable decisions, and whether they copy prompt content. Comparisons across tasks help establish where these behaviors recur. Under the baseline intensive-care prompt, MedGemma chose continued monitoring in all 450 attempts, achieving 33.3% accuracy. On the router, it used all six tools and answered 144 of 150 requests correctly. Meditron-7B produced unusable responses and copied prompt content across intensive-care monitoring and the non-medical evaluations at both four-bit and eight-bit quantization. MedGemma’s results motivate checking the medical instructions. Meditron’s recurring failures warrant checking the model configuration and how outputs are generated. Differences between tasks and mismatches between medical prompts and scoring rules limit conclusions about the causes of these failures. We recommend reporting selected actions, invalid and copied responses, and results for each prompt condition alongside accuracy.

Table reasoning remains challenging for large language models (LLMs), particularly in tasks that require multi-step inference over long and structured tables. Existing approaches predominantly rely on single-direction reasoning, which limits their ability to explore alternative hypotheses across tasks. In this work, we propose CRAFT, a unified Counterfactual Reasoning Framework that reformulates Tabular question answering and fact verification into a general bidirectional verification process. Our method explicitly constructs both declarative statements and their counterfactual variants. Evidence is then extracted from reasoning along both the original and counterfactual paths, and integrated via a weighted mechanism to arrive at the final answer. Experimental results show that our approach consistently surpasses representative baselines on table reasoning datasets such as WikiTQ and TabFact, achieving especially large improvements on complex question answering. Our framework also significantly mitigates performance gaps between different backbone LLMs. This indicates that counterfactual reasoning effectively overcomes the limitations of single-direction inference, guiding LLMs toward more discerning reasoning and establishing a more principled paradigm for structured reasoning tasks.

While Large language models (LLMs) have proved able to address some complex reasoning tasks, we also know that they are highly sensitive to input variation, which can lead to different solution paths and final answers. Answer consistency across input variations can thus be taken as a sign of stronger confidence. Leveraging this aspect, we introduce a framework where models are systematically pushed to diversify solution paths towards a final answer, thereby testing them for answer consistency across multiple input variations. Focusing on math problems, we induce variations in (i) order of shots in prompt, (ii) problem phrasing, and (iii) languages used. Experiments on a wide range of open-source LLMs of various sizes show that reasoning consistency differs by variation dimension, and that by aggregating consistency across dimensions, our framework enhances mathematical reasoning performance on monolingual datasets GSM8K and MATH500, and the multilingual dataset MGSM. We also discuss the different efficacy of the types of variations induced and how they could be further leveraged in future work.

Current evaluation methods for large language models are coarse-grained and decoupled from generation, producing generic explanations that fail to provide actionable feedback for model improvement. We propose CriticGen, a fine-grained, generation-aware evaluation framework that turns evaluation into actionable control for answer improvement. CriticGen first generates sample-specific evaluation dimensions and scoring criteria, then uses the induced rubric to jointly produce a score, a reason, an executable refinement suggestion, and a refined answer. Experiments show that fine-grained evaluation should be both instance-specific and actionable. CriticGen improves human-rated rubric relevance/coverage from 3.33/4.03 to 3.97/4.24. On the score-consensus-filtered RefineData test set, the best CriticGen variant reaches 0.9556/0.9560 Pearson/Spearman agreement with the reference scores and improves reason/suggestion F1 to 0.7554/0.7900. Independent validation provides additional support: on 180 human-scored instances, CriticGen-Qwen2.5-7B obtains higher observed score correlations than GPT-5, with 0.8868/0.8969 Pearson/Spearman versus 0.8542/0.8615. Crucially, its feedback translates into substantial answer improvement, raising accuracy from 45.36% to 91.26% on 5,000 examples with reliable gold answers, with 47.30% of answers corrected from Wrong→Right and only 1.40% degraded from Right→Wrong.

Large language models (LLMs) increasingly produce evaluative judgments, but rating-level human–model agreement does not establish whether they can replace human inputs in a downstream task. Using HEXACO personality items as a case study, we conduct a preregistered three-stage validation of whether LLM-generated social-desirability ratings can serve as inputs to forced-choice scale construction, a psychometric task that requires costly human ratings. In Stage I, three LLMs served as raters of item-level social desirability for sentence- and adjective-format items, with human ratings serving as references. In Stage II, human- and LLM-based ratings were entered into the same constrained optimization procedure. In Stage III, 400 participants completed the resulting forms, and latent trait estimates were compared using graded Thurstonian item response theory. LLM ratings showed strong item-level correspondence with human ratings (sentence: r = 0.942–0.963; adjective: r = 0.979–0.980), but optimized forms showed partial item overlap and sparse exact pair overlap. Despite these design differences, LLM-based forms produced latent trait estimates that closely matched those from human-based forms (mean r = 0.815 for sentence items and 0.768 for adjective items). These results support the conditional use of LLM-generated ratings as scalable construction inputs while showing why downstream validation is necessary.

Model merging is a practical way to combine several fine-tuned models, but merged models can lose performance relative to their task-specific counterparts. Task accuracy shows the degradation, but not how merging changes the model’s internal representations. We study this question through hidden-state distortion, which measures the difference between a merged model’s hidden states and those of the corresponding task-specific reference model on the same examples. Across two model families, six tasks, and three merge methods, we find that the distortion is highly anisotropic: much of the merged-reference change is concentrated in a small number of directions. Removing distortion components along dominant directions often improves accuracy more than removing those along unrelated directions. When we jointly fit a single distortion basis across tasks for each model family, this removal still improves accuracy, suggesting that the tasks share some useful distortion structure. In addition, the distortion score provides a label-free criterion for choosing among candidate task-arithmetic merge scales, matching supervised selection for both model families.

In-context learning (ICL) for multi-hop question answering (MHQA) depends critically on the choice of demonstrations. Yet most existing methods prioritize semantic similarity, even though semantically similar examples may follow different reasoning paths and therefore be poor demonstrations for the target query. We present STRIDE (STructural Reasoning for In-context Demonstration Extraction), a structure-centric framework for multi-hop QA. STRIDE first infers the latent reasoning structure of a query. It then performs positive–contrastive structural routing, selecting structurally compatible demonstrations as positives and semantically plausible but structurally mismatched examples as contrastives. Finally, it constructs structural prompts that expose compact, role-consistent reasoning traces over supporting evidence. Together, it turns demonstration retrieval from semantic matching into structure-aware problem-solving guidance. Empirical results show that STRIDE consistently improves downstream QA performance across settings.

Many artwork questions begin with vision but cannot be answered by visual recognition alone. A model may need to connect an artwork to its creator, depicted subject, collection, or location, and then follow multiple facts to reach the answer. We introduce ArtHopQA, a multimodal multi-hop question answering benchmark for artwork understanding, containing 4,885 image-grounded questions paired with supporting fact chains. Experiments with open and API large vision-language models show a large gap: even the strongest open model reaches only 36.87% accuracy when given the image and title, while gold-path performance reaches 97.77% to 99.80% when the supporting fact chain is provided. Text-only language models exhibit a similar gap, showing that the difficulty is not confined to visual input. Further analysis reveals high-prior geographic guesses, inconsistent answers under image-title conflict, and limited recovery of intermediate entities even when final answers are correct.

Large language models often lose track of persistent instructions and relevant information as multi-turn conversations grow. We study this cumulative contextual decay through three related failure modes: attention pollution, dilution, and drift. We propose **REA** (Role-aware Heuristic Episodic Attention), a context-management framework that assigns different persistence and representation policies to instructions and episodic interactions. Instructional Memory retains identified global constraints in a dedicated prefix. Episodic Memory preserves user inputs and compresses model replies, while heuristic retrieval selects raw text, compressed representations, or omission for each historical turn. On Long-MT-Bench+, REA improves the judge score from 6.32 to 7.36 on a 10-point scale, a 16.5% relative gain over the Vanilla baseline, and reduces average latency by 2.91×. Additional evaluations show aggregate gains on three backbones spanning 1.7B–7B parameters and on Chinese and English role-playing tasks. These results support role-aware context management as a practical approach to maintaining conversational continuity and instruction adherence.

Fact verification is a critical primitive for ensuring the reliability of Large Language Models (LLMs). While frontier LLMs are effective verifiers, their repeated long-context inference incurs substantial costs; meanwhile, existing specialized verifiers face a fundamental dilemma: label-only models suffer from black-box opacity, while free-form rationale generators, whether trained by distillation or explanation-oriented RL objectives, can produce fluent but not structurally checkable explanations. In this work, we propose AtomicCheck a Protocol-Guided Policy Optimization framework that trains specialized verifiers to produce protocol-guided, locally checkable traces, moving beyond opaque label-only predictions and free-form rationales. Rather than cloning teacher traces or optimizing unstructured explanations, AtomicCheck uses RL to make the verification protocol itself part of the policy objective. To preserve semantic integrity during atomic decomposition, we introduce a Contrastive Decomposition Reward that compares decomposed reasoning with a same-backbone holistic counterfactual and penalizes decompositions that reduce verification accuracy. Extensive experiments demonstrate that AtomicCheck outperforms existing specialized verifiers and approaches strong frontier LLMs while producing structured traces that expose local checks behind the final decision.

E-commerce conversations are increasingly multimodal, yet end-to-end response scores cannot localize which dialogue competence failed, and no public benchmark evaluates finegrained multimodal dialogue understanding for Korean commercial agents. We present KoMMERCE, a Korean multimodal fashion-domain benchmark that coordinates nine fine-grained dialogue understanding tasks on the same conversations: sentiment, intent, and strategy classification;image and text knowledge retrieval;dialogue state tracking; span detection; coreference resolution; and the first formulation of cross-modal dialogue relation extraction. The corpus contains 15,293 dyadic role-play dialogues under two-stage quality control. Baselines on six frontier multimodal LLMs reveal task-conditional patterns that should inform deployment choices: no model wins all nine tasks, and context length affects accuracy differently from one task to another.

Large language model (LLM) safety benchmarks often evaluate input risk, response selection, generation, and tool use separately, making comparisons across tasks difficult. We introduce KoEVD (Korean Ethical Violation Dataset), which links five evaluation and analysis tasks through annotated Korean source utterances: input-risk judgment, candidate-response selection, generation safety, response-strategy analysis, and non-executable mock-tool selection. The tasks share source IDs, with task-specific subsets and candidate sampling where required. Built through an approximately 3.5-year human-governed, AI-assisted process, KoEVD contains 13,552 curated user utterances and 71,395 linked response candidates. Across 12 open-weight and commercial models, observed task profiles differ, and aggregate scores obscure variation across severity, explicitness, utterance type, and topic. In matched generation and mock-tool-selection cases, responses classified as safe coexist with unsafe mock-tool choices; pooled unsafe-call rates are 9.73% and 9.31% for the classifier-safe and classifier-unsafe groups, respectively, with framing-specific differences. These diagnostic measurements do not establish real-world safety. KoEVD supports detailed Korean safety analysis through contextual annotations and source-linked comparisons.

Conventional prompting and prompt-based finetuning offer only weak semantic structure, leaving models to derive task-specific reasoning from limited cues. Existing techniques, such as using demonstrations, chain-of-thought prompting, or retrieval augmentation, can improve performance, but often at the cost of longer, more redundant prompts, higher inference overhead, and reliance on external knowledge sources. We present ConceptWeave, a novel concept-centric prompt-based finetuning approach built on adaptive semantic infilling. Instead of depending on external resources, it extracts concise, task-relevant concepts directly from a small portion of downstream data and organizes them into a dynamically evolving hierarchical memory. During fine-tuning, ConceptWeave identifies informative spans in the input through semantic alignment and augments them with corresponding concept representations, enabling fine-grained, localized conditioning of the model. As training progresses, the concepts are continuously refined and structured into a hierarchy that supports efficient retrieval while preserving semantic richness and scalability. We validate ConceptWeave across a range of NLP tasks, including misinformation detection, question answering, sentiment analysis, and hate speech classification, where it consistently outperforms strong baselines and recent state-of-the-art methods.

Large language models (LLMs) are now deployed worldwide, inspiring a surge of benchmarks that measure their multilingual and multicultural abilities. However, these benchmarks prioritize generic language understanding or superficial cultural trivia, leaving the evaluation of grounded tasks—where models must reason within real-world, context-rich scenarios—largely unaddressed. To fill this gap, we present CulturALL, a comprehensive and challenging benchmark to assess LLMs’ multilingual and multicultural competence on grounded tasks. CulturALL is built via a human–AI collaborative framework: expert annotators ensure appropriate difficulty and factual accuracy, while LLMs lighten the manual workload. By incorporating diverse sources, CulturALL ensures comprehensive scenario coverage. Each item is carefully designed to present a high level of difficulty, making CulturALL challenging. CulturALL contains 2,610 samples in 14 languages of 51 regions, distributed across 16 topics to capture the full breadth of grounded tasks. The experiments show that the best LLM achieve 44.48% accuracy on CulturALL, underscoring substantial room for improvement.

As Large Language Models (LLMs) continue to scale, inference efficiency has emerged as a critical bottleneck. Multi-Token Prediction (MTP) offers a promising avenue to accelerate LLM inference by predicting multiple future tokens simultaneously. Wherein, DeepSeek MTP has gradually become an indispensable component in industrial LLMs. However, it still faces two challenges: limited accept rates of MTP heads, and difficulties in jointly training multiple MTP heads. To address these challenges, we propose MTP-D, a simple yet effective self-distillation method that substantially boosts MTP head accept rates while maximally preserving main-head performance. We further introduce a looped extension strategy for MTP-D, enabling effective and economical extension to multiple MTP heads and yielding significant additional inference speedup (up to 4.55×). Beyond these contributions, we conduct systematic experiments across seven benchmarks to explore and validate key insights into distillation strategies and the scalability potential of DeepSeek MTP. Results demonstrate that MTP-D and the grouped extension strategy consistently enhance MTP head performance and inference efficiency, facilitating the practical usage of MTP in LLMs.

Multi-bit watermarking for large language models (LLMs) enables richer provenance information, but existing schemes remain unreliable for short texts because they rely on local or pre-specified allocation strategies. To address this limitation, we propose a Closed-Loop Error Feedback (CLEF) Multi-Bit Watermarking framework. By treating the LLM as a noisy channel, CLEF tracks the global feature state of generated tokens and dynamically measures its deviation from an ideal multidimensional target trajectory. The resulting dynamic error vector is then used to modulate the logits of candidate tokens via an inner-product scoring mechanism, biasing generation toward tokens that compensate for underrepresented payload dimensions. This closed-loop control enables robust multi-bit embedding with a simple, fully linear decoding process. Experiments on short-text generation show that CLEF improves bit extraction accuracy and complete message recovery rate over the evaluated multi-bit watermarking baselines. Furthermore, CLEF achieves higher bit extraction accuracy under different text perturbations while maintaining acceptable generation quality across downstream tasks such as text summarization and machine translation.

Pragmatic reasoning, including inferring intent, manipulation, and hidden meaning in dialogue, is essential for trustworthy language understanding yet remains underexplored in current large language models (LLMs). We introduce HIPO, a new benchmark designed to assess Hierarchical Pragmatic Tactic in multi-turn Conversations. Inspired by linguistic pragmatics, HIPO operationalizes a hierarchical annotation schema for multi-turn conversations and labels each utterance along one dialogue-level goal and three utterance-level dimensions: communicative intention (why), veracity strategy (how), and illocutionary act (what). To ensure reliable supervision, we design a structured generation pipeline to generate high-quality synthetic dialogues. The benchmark comprises over 4,088 benchmarking utterances with 6,350 contextual utterances across 1,131 dialogues drawn from 31 real-world-inspired scenarios. Benchmarking 22 state-of-the-art LLMs reveals a striking gap: while models achieve higher performance at recognizing surface speech acts(83.3% accuracy), they perform poorly on detecting veracity strategies(32.2%) and speaker intentions(45.1%). These results suggest current LLMs remain challenged by the pragmatic inferences operationalized in HIPO, particularly those involving information manipulation and intended effects beyond literal content. We further show that supervised fine-tuning on additional synthetic data generated by our pipeline can improve performance on HIPO, suggesting a practical path for targeted capability improvement within this benchmark setting.

Fake news detection is often treated as a content-centered task, yet social media posts are embedded in user reactions that express skepticism, endorsement, correction, ridicule, and affective responses. This paper asks what makes user reactions useful for fake news detection. We study performance according to different dimensions: text only, emotions in comments, and number of available comments. We study this question across Fakeddit and Weibo20 under both standard and temporal splits (in a temporal split, most recent posts are included in the testing split). Robustness of results is measured across two encoder families (BERT-base, RoBERTa-base) and two English emotion taxonomies (dair-ai/emotion, GoEmotions). Across these configurations, comment-only representations outperform post-only baselines in most evaluation regimes, though this gain depends on dataset, split, and number of available comments. Emotion-fine-tuned comment encoders produce a temporal-shift-specific significant benefit compared to non-temporal splits.

We compare post-only representations, generic comment representations, emotion-fine-tuned comment representations, and post–reaction fusion models, while varying comment availability, showing that a minimum number of comments is necessary to improve detection of fakes. In addition, we show that fine-tuning comment or reaction-encoders on relevant auxiliary tasks such as emotions has a significant impact on the distribution of embeddings of fake and real posts in their representation space.

In light of these findings, we believe that user responses—and their emotional structure for example—should be taken into account when detecting fake news or compiling datasets for this purpose.

Batch prompting is a practical inference strategy for large language models, but its safety implications remain underexplored. We show that the success of batch prompting for utility does not extend to safety: a harmful question that is reliably refused in isolation can elicit a harmful response when embedded in a batch of benign questions. We identify this as a distinct safety failure mode, not reducible to known vulnerabilities such as in-context learning or long-context effects—and analyze its causes from two complementary perspectives: alignment signal weakening and refusal signal dilution. Across widely used open-source and frontier commercial models, batch prompting consistently achieves high attack success rates as a simple black-box attack. We further show that batch-aware preference optimization effectively mitigates the vulnerability. These findings highlight a blind spot in current safety alignment and point to batch-aware alignment as a necessary step toward robust deployment.

Graduate program selection is a high-stakes decision where applicants struggle to identify programs matching their latent academic profiles. We address this as a retrieval problem: given a student profile, rank 531 graduate programs by estimated admission compatibility, i.e., how closely the student’s profile resembles each program’s historical admit pool. Because applicants rarely report rejections, training data consists only of positive labels, making negative supervision unavailable. We propose PURE (Positive-Unlabeled Retrieval Encoder), a positive-unlabeled biencoder with InfoNCE contrastive loss and a gated structural residual that fuses GPA and experience counts with free-text descriptions. With a novel multi-positive MRR protocol that corrects for false negatives, PURE achieves MRR=0.2696, outperforming BM25 (+4.9 points) and GPT-4o-mini (+8.3 points), while remaining statistically indistinguishable from Biencoder + GPT-4o-mini reranking at negligible cost.

Translation conveys meaning across languages, but does it preserve the personality a character conveys? We introduce an evaluation framework for quantifying how well translation systems maintain perceived character personality, the personality impression a reader forms from a character’s dialogue, described along the five Big Five dimensions; we use these dimensions as descriptive labels for the impression, not as a psychometric measurement of the character. Our framework applies personality predictors to both source and translated utterances, measuring cross-lingual consistency via Spearman rank correlation at the speaker level. We study entertainment subtitle translation, where every utterance is produced by a named character, and compare evaluation conditions that include, shuffle, or withhold the character names. We evaluate five MT systems, including dedicated MT APIs (Google Translate, DeepL) and LLMs (GPT-4o, Qwen3- 32B, Aya-expanse-32B), and additionally report human translation as a reference condition on GAME OF THRONES (English→12 target languages, 11 characters) and ONE PIECE (Japanese→English, 15 characters). Under our evaluation protocol, measured personality consistency varies across systems and language pairs, and high BLEU/COMET translation quality does not reliably imply high personality consistency.

Extracting sentence representations from pretrained large language models (LLMs) is a training-free approach that leverages their inherent semantic understanding. Existing methods primarily rely on prompt engineering to distill core semantics, but the resulting representations often retain language-specific features that hinder semantic tasks. Moreover, these approaches fail to exploit the cross-lingual capabilities of LLMs, remaining confined to monolingual settings. To address these limitations, we propose CLSF, a novel method that actively introduces diverse language perturbations when encoding the same sentence and then fuses the perturbed embeddings, effectively reducing language-specific biases and producing a more universal representation. CLSF is plug-and-play and can be integrated with existing prompt-based methods. Experiments on Semantic Textual Similarity (STS), multilingual alignment, and various downstream tasks show that CLSF consistently improves performance across different LLMs.

Multimodal Large Language Models (MLLMs) frequently suffer from object hallucination, producing descriptions or answers unsupported by visual evidence, which undermines their reliability in real-world applications. This paper begins with an intriguing phenomenon: prefacing queries with statements explicitly contradicting image content often yields more accurate responses than direct questioning. We term this the Contradictory Prompting Effect (CPE). Priming inputs with assertions opposing reality triggers an intrinsic self-verification mechanism within MLLMs, where conflicting claims compel the model to prioritize visual evidence over linguistic priors. Inspired by this discovery, we propose Contradiction-Guided Visual Reasoning (CGVR), a novel training-free framework for hallucination mitigation. CGVR systematically generates deliberate contradictory captions alongside standard ones. These captions drive a dual-channel inference process, in which conflicting signals enforce rigorous cross-checking against visual evidence. Our work reveals MLLMs’ latent self-verification capability triggered by contradictions and establishes a paradigm for lightweight, reliable multimodal reasoning. Extensive evaluations on POPE and MME benchmarks demonstrate that our proposed CGVR significantly mitigates hallucinations across diverse MLLMs. Our code is available at https://github.com/dhr1698/CGVR.

We present a Korean Benchmark Suite (KBS) comprising Ko-ARC, Ko-GSM8K, Ko-EQ-Bench, Ko-WinoGrande, Ko-LAMBADA, and Ko-IFEval, covering six capabilities across 9,396 items. The suite combines review, localization, direct Korean construction, and redesign for Korean, avoiding a translation-only approach to benchmark adaptation. We evaluate Korean-developed models, multilingual open models, and closed API models under protocols that account for model access, prompt format, and scoring differences. Results show that Korean benchmark scores depend on task construction, scoring rules, prompting, and model post-training. These findings suggest that Korean evaluation should report both final scores and the settings used to obtain them.

Automated Knowledge Base Construction (AKBC) is a core NLP task, and recent work proposes generating knowledge bases directly from large language models (LLMs), treating the model itself as the knowledge source. However, LLMs natively possess no representation of entities, leading to duplicate entries as well as conflations.

We propose DisamKB, a methodology for constructing disambiguated KBs directly from LLMs. DisamKB incorporates on-the-fly disambiguation of entities, relations and classes, and is meticulously designed to satisfy both scalability and disambiguation accuracy. We analyze the central design decisions and characterize the trade-offs between accuracy, scale, and cost. We execute DisamKB at scale, obtaining a materialized KB containing over 1M disambiguated entities and 38.4M triples. This represents the first million-scale LLM-native KB with explicit internal canonicalization of entities, relations, and classes, a significant departure from prior Wikimedia-centric works. DisamKB is anonymously available at https://gptkb.org/.

Group-relative policy optimization (GRPO, DAPO, ARPO) has become the dominant recipe for RL fine-tuning of reasoning LLMs, yet in the resource-constrained regime most practitioners train in (a single GPU, small batches) it consistently suffers from rapid entropy collapse that kills exploration, together with sparse credit assignment in long rollouts, which is most extreme under multi-turn tool use but already present in long single-turn chain-of-thought. We address both failure modes with JANUSPO, a drop-in GRPO modification that introduces asymmetric PPO clipping 𝜖₊<𝜖_- to tolerate larger downward than upward updates, pairs it with an asymmetric KL schedule 𝛽_pos<𝛽_neg that doubles as a free entropy regulator without any explicit upper bound, and redistributes the terminal reward through a turn-weighted decomposition w_t(n) that generalises naturally from single-turn CoT to multi-turn rollouts. On Qwen2.5-3B-Instruct / GSM8K, JANUSPO improves from 74.8% to 84.3±0.2% (mean±std over 2 seeds) across 14 controlled ablations, and the same recipe lifts 7B from 85.3% to 93.6% with no hyper-parameter retuning. On multi-turn DeepSearch (103 GAIA + 100 HLE), SFT-warmup-then-JANUSPO is the only RL recipe whose paired comparison against SFT survives best@8 outlier verification, achieving strict dominance on all four reward and F1 columns while a matched-init GRPO baseline’s apparent F1 wins collapse to 0%. Beyond this main result, we document a systematic negative finding: SFT warm-up is a necessary precondition for GRPO-style multi-turn RL on format-unaware base models.

Multimodal aspect-based sentiment analysis (MABSA) aims to identify aspect-level sentiments by jointly modeling textual and visual information, which is essential for fine-grained opinion understanding in social media. Existing approaches mainly rely on discriminative classification with complex multimodal fusion, yet lacking explicit sentiment explainability. In this paper, we reformulate MABSA as a generative and explainable task, proposing a unified framework that simultaneously predicts aspect-level sentiment and generates natural language explanations. Based on multimodal large language models (MLLMs), our approach employs a prompt-based generative paradigm, jointly producing sentiment and explanation. To further enhance aspect-oriented reasoning capabilities, we propose a dependency-syntax-guided sentiment cue strategy. This strategy prunes and textualizes the aspect-centered dependency syntax tree, guiding the model to distinguish different sentiment aspects and enhancing its explainability. To enable explainability, we use MLLMs to construct new datasets with sentiment explanations to fine-tune. Experiments show that our approach not only achieves consistent gains in sentiment classification accuracy, but also produces faithful, aspect-grounded explanations.

As ideation agents become more capable, evaluating them poses an increasing challenge. LLM-as-a-judge approaches are brittle and gameable, while human evaluation is expensive and hard to scale. To address this, we introduce HorizonBench, an evaluation methodology leveraging future scientific discoveries. HorizonBench uses published papers as ground truth for ideation quality. Specifically, ideation agents pretraining and context retrieval data are strictly constrained to pre-publication knowledge, and scored on how well their predicted ideas align with subsequently published work. Using HorizonBench, we conduct the first unified comparison of five published ideation agents across multiple frontier model backbones. We verify that our soft recall metric captures meaningful ideation quality aligned to human judgement via a user study with 51 domain experts. We then uncover a surprising finding: a naïve baseline that receives only the research goal as an input consistently matches or outperforms all existing agents that require retrieval or multi-step reasoning, highlighting the importance of the research goal for ideation. Based on this result, we develop OSCAR, an ideation agent that distils retrieved papers into insights and explicitly reinforces the research goal in each step of the pipeline. OSCAR outperforms all evaluated agents on HorizonBench. We publish the code and an online version of HorizonBench.

Creative text is inherently subjective: any opinion can be equally valid, and there is no objective ground truth. The only way to quantify such subjectivity is to model the crowd of opinions that produces it. Large language model (LLM)-as-a-judge has emerged as a scalable substitute for human raters, but whether an LLM can stand in for the human crowd on AI creativity remains unexplored. We develop a decomposition framework (four main components, 11 sub-components) and a three-step rating procedure, applied identically to human raters and to different LLM judging conditions on AI-generated stories. Humans rate AI creativity with the same subjectivity that creativity research documents for human-authored creative work. Across three structural comparisons (disagreement on the same story including ranking; revision from impressionistic to reflective; sub-component weighting of the final rating), no LLM reproduces all three human patterns. We frame the contribution as a methodological framework plus a quantification of where LLM judges currently fall short of the human ratings in our study. We release all stories, ratings, and code at https://github.com/ku-nlp/creativity-evaluation-artifact.

Retrieval-augmented generation (RAG) over knowledge graphs requires retrievers that can effectively capture both graph structure and semantic information. Recent approaches have explored graph neural network (GNN)-based retrievers to model graph topology in multi-hop reasoning tasks. In parallel, graph language models (GLMs) have emerged as a promising paradigm that integrates graph reasoning and the semantic capabilities of language models. In this work, we introduce a GLM-based retriever and investigate the comparative strengths of GLM-based, GNN-based, and traditional vector-search-based retrievers in single- and multi-hop RAG settings, and with a particular focus on transferability to unseen domains. Our findings suggest that finetuned GLM retrievers generalize better out of domain, achieving SOTA on two multi-hop benchmarks. On in-domain multi-hop QA datasets they remain comparable to prior work, with promising scaling as parameters and subgraph coverage increase. GNN-based retrievers achieve higher graph coverage with an efficient training setup, whereas the vector-search baseline excels at single-hop datasets.

The evolution of AI systems toward agentic operation and context-aware retrieval necessitates transforming unstructured text into structured formats like tables, knowledge graphs, and charts. While such conversions enable critical applications from summarization to data mining, current research lacks a comprehensive synthesis of methodologies, datasets, and metrics. This systematic review examines text-to-structure techniques and the encountered challenges, evaluates current datasets and assessment criteria, and outlines potential directions for future research. We also introduce a universal evaluation framework for structured outputs, establishing text-to-structure as foundational infrastructure for next-generation AI systems.

Agents using the Model Context Protocol (MCP) rely on semantic matching to select tools from third-party servers, exposing a semantic supply-chain risk through attacker-controlled metadata and outputs. We introduce A2M (Attraction-to-Manipulation), a two-stage black-box framework for hijacking MCP agents. The Attraction phase optimizes tool metadata to increase invocation probability; the Manipulation phase uses execution traces to refine adversarial tool returns that steer agents toward attacker-desired outcomes. On LiveMCPBench, direct attacks optimized and evaluated on GLM-4.6 achieve a macro-average malicious tool invocation rate of 93.6% across four scenarios, increase weighted token costs to 32.4× the benign baseline under Cognitive Denial of Service, and attain a mean attack success rate of 74.4% across Information Exfiltration, Environment Integrity Compromise, and Reasoning Derailment. Transfer to four other models without re-optimization yields corresponding macro-averages of 63.6%, 2.7×, and 24.5%. These findings motivate stronger tool vetting and runtime isolation in MCP ecosystems. Code is publicly available at https://github.com/Lilaizhen/A2M.

LLM-as-a-Judge is the standard paradigm for NLP evaluation, yet its systemic reliability remains poorly understood despite being widely treated as a deterministic ground truth. We present a comprehensive reliability audit, stress-testing six frontier models across four benchmarks, five prompt formats, two presentation orders, three sampling temperatures, and ten repetitions per condition. Our empirical analysis reveals severe vulnerabilities: verdicts change across identical replications at temperature zero, position-order swaps flip the majority of verdicts on challenging tasks, and the most deterministic judge achieves perfect consistency by trivially repeating incorrect verdicts, agreeing with ground truth only 51% of the time. To formalize these multi-faceted failure modes, we introduce the trustworthy verdict rate (T), a unified metric capturing the joint probability that an evaluation is reproducible, order-invariant, and accurate. Using T , we derive a theoretical upper bound on accuracy imposed by position bias and show that reliability is item-specific rather than model-level. Finally, we demonstrate that shifting from pairwise win-rate to holistic rubric scoring improves trustworthiness more than any single-format prompting intervention, offering an actionable framework for robust NLP evaluation.

Hallucination, the generation of non factual content by AI systems, poses a serious risk in medical contexts, where errors can directly affect patient outcomes. We present MedHal, a novel large-scale dataset specifically designed to assess capabilities and train models on the task of hallucination detection in medical texts. Current hallucination detection methods face significant limitations when applied to specialized domains like medicine, where they can have disastrous consequences. MedHal addresses this issue by incorporating diverse medical text sources and tasks covering both intrinsic and extrinsic hallucinations, and by providing a substantial volume of data samples suitable for training medical hallucination detection models. We demonstrate MedHal’s utility by training and evaluating a baseline medical hallucination detection model, showing improvements over general-purpose hallucination detection approaches. This resource enables more efficient evaluation and training of medical text generation systems while reducing reliance on costly expert review, potentially accelerating the development of medical AI research.

Deploying large language models in production often requires constraining outputs to structured formats such as JSON or XML, and prior work treats the resulting accuracy loss as an inherent "structure tax". We re-examine this claim by evaluating a battery of models, datasets and schemas, measuring task accuracy, confidence calibration, and hidden-state geometry. The tax turns out to depend on schema design rather than on structure per se: reasoning-first field ordering matches or exceeds free-form accuracy, while answer-first ordering causes steep drops, particularly in smaller models. Format sensitivity scales inversely with a task’s own structural constraints, and schemas that preserve reasoning order also improve calibration with CKA showing greater separability between correct and incorrect representations in middle transformer layers. Our findings indicate that properly designed structured formats can match or exceed free-form performance, reframing the critical question from ’whether to structure’ to ’how to structure’ for optimal reasoning preservation.

Large vision-language models (LVLMs) suffer from visual ungroundedness: they can produce a fluent, confident, and even correct response driven entirely by language priors, with the image contributing nothing to the prediction. Existing confidence estimation methods cannot detect this, as they observe model behavior under normal inference with no mechanism to determine whether a prediction was shaped by the image or by text alone. We introduce BICR (Blind-Image Contrastive Ranking), a model-agnostic confidence estimation framework. The LVLM stays entirely frozen; instead, we extract its hidden states twice per sample, once with the real image-question pair and once with the image blacked out while the question is held fixed. A small, separate probe is then trained on the real-image hidden state, regularized by a ranking loss that penalizes higher confidence on the blacked-out view, teaching it to treat visual grounding as a signal of reliability. At inference only the single real-image pass is used, so BICR adds zero cost beyond a standard single-view probe. Evaluated across five modern LVLMs and seven baselines on a benchmark covering visual question answering, object hallucination detection, medical imaging, and financial document understanding, BICR achieves the best cross-LVLM average on both calibration and discrimination simultaneously, with statistically significant discrimination gains robust to cluster-aware analysis, at 4–18× fewer parameters than the strongest probing baseline.

LLM agents accumulate interaction histories that exceed their context windows, motivating extensive research on memory systems. Yet most existing designs are tailored to a single scenario (such as multi-session chat or a specific trajectory format), offering little evidence that they generalize across the heterogeneous trajectories encountered in deployment. We evaluate twelve baselines spanning specialized memory systems, long-context models, and an unindexed search agent across five scenarios: single-turn QA, multi-session chat, agentic-trajectory QA, memory stress tests, and long-horizon agentic tasks. Notably, DCI-Lite, an unindexed baseline that searches raw text via grep and bounded reads, is among the strongest existing baselines. This suggests that memory performance depends partly on granting the agent active control over retrieval rather than coupling a static store to a single retrieval route. We instantiate this diagnostic insight in **AutoMEM**, a strong baseline that exposes both raw-text and indexed memory views through an agent-controlled interface. Across the evaluated systems, AutoMEM achieves the best mean cross-scenario rank.

Large Language Models (LLMs) can memorize and reproduce sensitive, copyrighted, or otherwise undesirable training content, creating privacy, safety, and regulatory concerns. Machine unlearning offers a practical alternative to full retraining, but many existing methods apply broad or fixed parameter updates that can degrade utility and remain brittle under deployment changes such as post-training quantization, where forgotten knowledge may partially re-emerge.

We propose Forgetting Only What Matters via Unlearning Layers (FOM-UL), a layer-level unlearning framework that selects transformer layers using a forget-to-retain significance score. This score identifies layers with high influence on the forget set and low sensitivity to the retain set, allowing FOM-UL to concentrate updates where they are most effective while leaving most of the model unchanged. This targeted update strategy improves the forgetting-utility trade-off and provides an empirical path toward quantization-resilient unlearning by reducing the chance that small, diffuse updates are erased by low-bit rounding.

Across TOFU, KnowUnDo, and MUSE-style evaluations, FOM-UL reduces residual memorization compared with strong GA, NPO, KLD, SURE, ReLearn, and LUNAR-based baselines while preserving retain-set utility close to the vanilla model. Under 8-bit and 4-bit post-training quantization, FOM-UL maintains a favorable forgetting-utility trade-off, combining competitive memorization suppression with stronger retained utility. Overall, FOM-UL provides an efficient unlearning strategy that improves targeted forgetting, utility preservation, and deployment robustness without claiming formal guarantees of erasure.

Grokking, the phenomenon in which neural networks generalize long after fitting their training data, has been studied in supervised settings on many epochs. LLM pre-training instead involves next-token prediction over an unlabeled corpus, with limited data repetition and no explicit train/validation split. To address this, we propose an exposure-based framework that enables the study of grokking-like dynamics during LLM pre-training. We ground our evaluation in BLiMP minimal pairs, which provide controlled grammatical contrasts. For every BLiMP minimal pair, we identify a critical phrase, the smallest continuous span that captures the grammatical contrast and the phenomenon-relevant context. Examples whose critical phrase appears in the pre-training window are assigned to the proxy-train split; the remaining examples are assigned to the proxy-validation split. Across five grammatical phenomena, we observe delayed generalization. Analyzing pre-training checkpoints before and after generalization shows that grammatical concept vectors become more predictive of grammatical acceptability and occupy a higher-dimensional subspace after generalization. We also find that attention from the critical token to the relevant context token is concentrated in a small number of heads.

Improving tool use in large reasoning models (LRMs) typically demands expensive test-time compute, training resources unavailable to API-only practitioners, or expensive iterative rollouts with execution feedback required by training-free context-engineering methods, all incompatible with resource-constrained deployment. We introduce **GEAR**, a training-free framework that extracts compositional rules from tool-use failures via single-pass meta-reasoning and distills them into inference prompts. We hypothesize that combining explicit rules with examples outperforms either alone. Across 78 API domains, **GEAR**+2-shot outperforms 3-shot by 24.9% on Qwen3-30B, validating our complementarity hypothesis. On Qwen3-4B, **GEAR** exceeds LoRA SFT by 24.3% and RAG-3-shot by 38.5%, while **GEAR**+2-shot beats the state-of-the-art context-engineering method ACE by 38.8% at 4.1× lower cost. Remarkably, Qwen3-30B with **GEAR**+2-shot nearly doubles zero-shot accuracy while remaining under 1 s latency, demonstrating rollout-free adaptation for resource-constrained deployment.

Recent advances in large language models (LLMs) have accelerated automated scientific discovery (ASD), but existing ideation tools still struggle to generate hypotheses that are both testable, literature-grounded, and adaptive to prior experimental outcomes. We developed HARPA to address these challenges by following a human-inspired ideation workflow that mines literature trends, explores hypothesis design spaces, and converges on precise, testable hypotheses by pinpointing research gaps and justifying design choices. Human evaluations show that HARPA-generated hypothesis-driven research proposals performs comparably to a strong baseline system, AI-researcher, in novelty and overall quality, while showing a positive trend in feasibility (+0.78, p<0.05, bootstrap) and a significant gain in groundedness (+0.85, p<0.01, bootstrap) on a 10-point Likert scale. When evaluated with the ASD agent CodeScientist, HARPA produced more successful executions (20 vs. 11 out of 40) and fewer failures (16 vs. 21 out of 40) indicating that expert feasibility judgments track execution success. HARPA also learns a reward model that scores hypotheses based on prior experimental outcomes, achieving approx. a 28% absolute gain over HARPA’s untrained baseline scorer.

Foundation-model agents in multi-step, open-ended environments frequently suffer from compounding errors, where early mistakes contaminate long-horizon trajectories. While Multi-Agent Debate (MAD) succeeds in deterministic domains with external validators (CITATION), agents in subjective, non-deterministic tasks such as persuasion experience problem drift and sycophantic conformity. Focusing on subjective argumentation, we identify semantic leakage in standard Retrieval-Augmented Generation (RAG) as a reproducible trigger for these failures: standard RAG prioritizes vocabulary overlap over logical structure.

To mitigate this leakage, we introduce Taxonomic Strategy RAG (TS-RAG), a systems intervention that routes strategy retrieval through a discrete categorical bottleneck to decouple argumentative structure from topical content. Zero-shot, cross-domain evaluations on r/ChangeMyView show that TS-RAG improves the transfer of abstract argumentative logic where standard semantic retrieval degrades (+14.5 win-rate points over an unaugmented persuader for Gemini Flash 3.0, p<0.05). In asymmetric deployments that mix lightweight and frontier models, TS-RAG raises the win rate of a lightweight persuader against a stronger opponent (70.5% → 78.5%, p<0.05) and reduces rounds to consensus when the persuader is already dominant. Finally, we introduce trace-level diagnostics via a turn-by-turn Debate State Representation (DSR), and show that strict stance-holding constraints on the opponent are necessary to prevent evaluation collapse caused by default agentic sycophancy.

Listwise passage reranking often achieves strong ranking performance, but it incurs substantial computational cost. We propose FLASH, an efficient listwise reranking framework based on semantic token compression and single-step listwise scoring. FLASH represents each query–passage pair with a small set of query-aware semantic tokens and jointly scores all candidates in a single decoder step, enabling query-conditioned joint scoring without expensive token-level listwise processing. We further introduce multi-cardinality training, which allows a single model to support different semantic-token budgets at inference time. We also propose FLASH-LITE, a lightweight variant with offline passage encoding for low-cost online inference. Across TREC and BEIR benchmarks, FLASH improves average ranking effectiveness at comparable online latency and slightly lower FLOPs, while FLASH-Lite substantially reduces query-time cost with a modest effectiveness trade-off. FLASH-LITE enables offline passage encoding, substantially reducing online query-time cost while maintaining competitive reranking performance. Code is provided in the supplementary material.

With the telecommunications field embracing zero-touch management alongside novel O-RAN and AI-RAN frameworks, contemporary telecom networks now function as immensely intricate and heavily softwarized codebases. While automated software engineering (ASE) tools and Software Engineering (SWE) Agents hold the potential to alleviate the critical code generation bottleneck in this domain, their ability to navigate and modify specialized, mathematically rigorous wireless stacks like srsRAN 5G remains unverified. General-purpose coding benchmarks fail to capture the stateful logic and strict requirements of telecommunications, leaving a critical evaluation gap. In this paper, we introduce TeleSWEBench, the first commit-driven benchmark specifically designed to measure an agent’s performance in the telecom domain. We mine real developer commits from the srsRAN 5G repository and distill them into structured test cases across three difficulty tiers (Easy, Medium, and Difficult). Our benchmark consists of 734 questions that are accompanied by executable unit tests. To avoid the rigidity of test cases, we further propose a hierarchical LLM-as-a-Judge framework called TeleJudge that scores agent outputs at the file level and aggregates verdicts holistically. This follows an evaluation based on context and semantic similarity in parallel to a standard unit test-based evaluation. Using this benchmark, we evaluate AIDER, OpenHands, and the Claude Code frameworks, powered by state-of-the-art reasoning LLMs, including Qwen, GPT-OSS, Gemma 4, Kimi, GLM, and Qwencoder 2.5. Our two-stage evaluation reveals that models suffer from a lack of both localization accuracy and functional correctness, with the strongest ASE tools achieving up to 25% of shippable changes.

Large Language Model (LLM) personalization holds great promise for tailoring responses by leveraging personal context and history. However, real-world users usually possess sparse interaction histories with limited personal context, such as cold-start users in social platforms and newly registered customers in online E-commerce platforms, compromising the LLM-based personalized generation. To address this challenge, we introduce GraSPer (Graph-based Sparse Personalized Reasoning), a novel framework for enhancing personalized text generation under sparse context. GraSPer first augments user context by predicting items that the user would likely interact with in the future. With reasoning alignment, it then generates texts for these interactions to enrich the augmented context. In the end, it generates personalized outputs conditioned on both the real and synthetic histories, ensuring alignment with user style and preferences. Extensive experiments on three benchmark personalized generation datasets show that GraSPer achieves significant performance gain, substantially improving personalization in sparse user context settings.

For the practical deployment of the Role-Playing Agents with Large Language Models, prior work has mainly focused on suppressing out-of-character behavior relative to explicitly provided character information such as personas. However, as shown in psychological research, out-of-character behavior can arise not only from contradictions with explicit information but also from mismatches with character images implicitly inferred by users. Modeling user-inference-dependent out-of-character behavior is a challenging task, because it is unclear what kinds of inferences users make from existing information. To tackle this problem, we construct a dataset of 480 dialogue scenarios and collect 2,152 descriptions from users familiar with those titles. Using cases that annotators judged inappropriate despite the absence of contradiction with existing settings, we derive a 12-category typology, organized into general, title-specific, and individual-specific groups. We also conduct an exploratory evaluation of category-guided feedback, finding no consistent overall advantage over standard self-refinement but possible benefits for some characters. Our shareable code and dataset are at https://github.com/CyberAgentAILab/User-Inference-OOC.

Discovering student misconceptions is a key challenge in intelligent tutoring systems, where identifying recurring error-producing patterns can support targeted feedback and adaptive practice. In settings without a predefined misconception taxonomy, clustering is a natural approach for discovering such patterns at scale. Yet clustering is only as good as its representation: full student records contain problem wording, numerical context, and solution-specific details that embeddings can conflate with the student’s error, obscuring the misconception behind the wrong answer. We construct an LLM step-error-annotation-first pipeline that generates step-error annotations, embeds them with or without problem and student response text, and clusters the resulting representations. On MalruleLib, a 101-class misconception benchmark, clustering step-error annotations alone improves clustering accuracy by 0.22 and normalized mutual information by 0.05 on average over clustering the problem and student response, consistent with annotations capturing recurring error patterns more effectively. On Eedi and unlabeled MalAlgoQA, annotation-based representations show higher average held-out clustering accuracy and internal cluster separation, respectively, than their unannotated baselines. To help experts interpret these clusters, the pipeline then generates candidate misconception descriptors and coherence scores for human triage. Overall, these findings show that input representation substantially affects clustering results and that step-error annotations often improve clustering quality.

Large language models (LLMs) are strong sequence predictors, but standard context accumulation offers no explicit mechanism for revising the memory carried between predictions. We propose LLM-as-RNN, an inference-time framework for online learning with frozen LLMs through a bounded natural-language memory state. At each timestep, the model predicts from the current observation and its memory, receives feedback, and rewrites the state to correct errors and retain task-relevant patterns without updating model parameters. Implemented as a structured system prompt, this state makes the adaptation process inspectable through human-readable memory traces. We evaluate LLM-as-RNN on sequential prediction benchmarks in healthcare, meteorology, and finance across Llama, Gemma, and GPT-oss model families. Comparisons with static-context and feedback-aligned baselines examine predictive performance, feedback sources, memory budgets, and inference cost. Our results demonstrate that feedback-driven memory rewriting supports online error correction and sequential prediction under a fixed memory budget, with performance varying across tasks and backbones.

Expert writers calibrate their claims with qualifiers such as “likely benign“ in a radiology report, “may constitute“ in a court ruling, and “suggest“ in a scientific paper. We ask whether this calibration survives when Large Language Models (LLMs) summarize and rewrite professional text. We define epistemic erosion as any change in a source’s expressed certainty, including qualified claims that are dropped or introduced without basis. We measure it with the Epistemic Faithfulness Score (EFS), computed by aligning source and output claims one-to-one with rule-based matching and assigning each distortion to one of six types, namely escalation, attenuation, omission, fabrication, scope shift, and severity warp. On clinical notes, mean EFS for 11 open-weight LLMs ranges from 0.415 to 0.493; three of these models are also evaluated on legal and scientific text. Four hosted models (GPT-4o, GPT-5.4, Phi-4, Mistral-Large-3) reach at most 0.483 in any domain. The most frequent detected distortion is omission (66%), in which a qualified claim is dropped entirely; claims that survive tend to shift toward greater certainty. Model scale and medical adaptation bring no consistent gain, seven prompting strategies bring at most modest gains, and ROUGE-L and BERTScore correlate only weakly with EFS (|r| ≈ 0.1).

Verifier-guided best-of-N decoding has two stages: a generator samples candidate chains of thought, and a verifier selects among them. A subgroup disparity may already be present in the generator’s candidates or be amplified during selection. We separate the two stages using adversarial demographic counterfactuals of medical exam questions. On HuatuoGPT-o1-8B, a substantial race accuracy gap is already present in a single sampled response, before verifier selection. Across seven generators, only Gemma-3-27B-it under Skywork-RM shows significant verifier-induced amplification after multiple-testing correction. Even an oracle selector leaves a residual gap on every medical generator analyzed, showing that correct answers are less often available for one subgroup. Counterfactually paired low-rank adaptation (LoRA) of the generator reduces the single-sample gap on two medical models, although on HuatuoGPT-o1-8B the reduction does not persist after self-judge selection. Single-variant controls indicate that the single-sample reduction is not explained by additional training steps alone. A short fairness instruction modestly reduces the single-sample gap, and a direct preference optimization (DPO) variant of the paired training also reduces it and, on HuatuoGPT-o1-8B, keeps race mention near its baseline rate. For the medical generators studied, candidate generation rather than inference-time selection is the primary locus for mitigating this gap.

Diffusion language models (DLMs) decode several tokens per step. However, the grammar-constrained decoders evaluated here, IG-CD and LAVE, verify candidate tokens sequentially within each denoising step and treat every violation as a rejection, a lookahead verification, or a fallback to autoregressive decoding. We observe that a violation is local to the first token the parser cannot consume, so the rest of the parallel step can be kept. We propose Dgrammar, a training-free decoder that repairs that frontier token in place under the same forward-pass logits, paired with a grammar-gated backoff schedule for the unmask batch size, asynchronous mask construction, and a bounded grammar-guided tail. On JSONSchemaBench medium with LLaDA-8B, Dgrammar improves schema validity by 8.6 percentage points over the strongest DLM baseline while reducing mean latency by 5.8x and eliminating timeouts. Validity and latency gains also hold on the other JSB splits and on Dream-7B, and experiments with a SMILES grammar demonstrate cross-grammar transfer.

In recent years, multimodal large language models have achieved substantial progress, yet their performance remains limited in specialized domains such as circuit diagram understanding. A key bottleneck lies in the lack of diverse and large-scale training data tailored to the structural characteristics of circuit diagrams, together with the high cost of expert annotation. To address this challenge, we propose CircuitPile, a systematic data synthesis framework for circuit visual-language modeling. CircuitPile integrates heterogeneous circuit resources from multiple sources and constructs standardized label spaces and multi-granularity visual representations. Based on this framework, we build 738K domain-specific pre-training samples and 100K task-oriented instruction-tuning samples for the pre-training and supervised fine-tuning stages, respectively. Furthermore, to systematically evaluate fundamental visually grounded circuit perception, we introduce CircuitPerception-Bench, a diagnostic benchmark comprising 2,700 multiple-choice samples across three dimensions: component recognition, spatial understanding, and quantitative reasoning. CircuitPerception-Bench focuses on fundamental visually grounded perception and complements external benchmarks such as AMS Benchmark. Experimental results show that multimodal models trained with CircuitPile achieve significant performance gains on the internal benchmark and demonstrate favorable transferability on the external public AMS Benchmark. For example, applying CircuitPile-based SFT to Qwen3-VL-8B improves the scores on CircuitPerception-Bench and AMS Benchmark from 0.5759 and 0.5067 to 0.7703 and 0.5988, respectively.

Linear probes on language-model hidden states can predict output correctness, but it remains unclear whether they capture transferable correctness representations. We study out-of-distribution correctness-probe transfer across seven datasets, four task families, and three open-weight LLMs: Qwen3-8B, Llama-3.1-8B, and Gemma-2-9B, with all findings validated across five random seeds. Within-family transfer is consistently stronger than cross-family transfer (gap 0.075 AUC), with code generation showing the clearest pattern across MBPP-test, MBPP-train, and HumanEval. We further find that probe weight directions are nearly orthogonal across most dataset pairs despite strong predictive transfer, suggesting that probes do not encode a single canonical correctness direction. Comparing probes to logprob confidence baselines, we find probe advantage is concentrated in in-domain and within-family transfer cells. We also identify a layer-selection confound where early-layer shortcuts can appear transferable in-domain but fail cross-source. Overall, correctness probes transfer selectively rather than universally, making them most useful in-domain and within task families.

Large language models (LLMs) exhibit strong performance on self-contained programming tasks. However, they still struggle with repository-level software engineering (SWE), which demands (1) deep codebase navigation with effective context management for accurate localization, and (2) systematic approaches for iterative, test-driven code modification to resolve issues. To address these challenges, we propose SWE-Adept, an LLM-based two-agent framework where a localization agent identifies issue-relevant code locations and a resolution agent implements the corresponding fixes. For issue localization, we introduce agent-directed depth-first search that selectively traverses code dependencies. This minimizes issue-irrelevant content in the agent’s context window and improves localization accuracy. For issue resolution, we employ adaptive planning and structured problem solving. We equip the agent with specialized tools for progress tracking and Git-based version control. These tools interface with a shared working memory that stores code-state checkpoints indexed by execution steps, facilitating precise checkpoint retrieval. This design enables reliable agent-driven version-control operations for systematic issue resolution, including branching to explore alternative solutions and reverting failed edits. Experiments on SWE-Bench Lite and SWE-Bench Pro demonstrate that SWE-Adept consistently outperforms prior approaches in both issue localization and resolution, improving the end-to-end resolve rate by up to 4.3%.

Autoregressive (AR) models have recently shown strong performance in image generation, where a key component is the visual tokenizer (VT) that maps continuous pixels into discrete tokens. The quality of the VT largely determines the upper bound of AR model performance. However, existing discrete VTs significantly lag behind continuous variational autoencoders (VAEs), leading to degraded reconstructions and poor preservation of fine details and text. Current benchmarks primarily evaluate end-to-end generation quality, making it difficult to isolate VT performance. We introduce VTBench, a comprehensive benchmark for systematic evaluation of visual tokenizers across three core tasks: image reconstruction, detail preservation, and text preservation. Evaluating state-of-the-art VTs with multiple metrics, we find that continuous VAEs consistently outperform discrete VTs, especially in retaining spatial structure and semantic details. Discrete VTs often produce distorted reconstructions and fail to preserve fine textures and text. We release VTBench publicly to facilitate future research on robust, general-purpose visual tokenizers.

The rapid proliferation of mobile applications has resulted in a vast volume of user-generated reviews, which often contain critical insights into bugs, feature demands, and overall user satisfaction. Efficiently organizing and interpreting this feedback is essential for developers seeking to enhance app quality and user experience. In this study, we propose a fine-grained, multilabel classification schema for app reviews, structured as a two-level taxonomy. The first level categorizes reviews into three primary classes: Bug Report, Feature Request, and App Satisfaction Feedback. The second level further refines these categories into subtypes addressing specific concerns such as performance, usability and privacy. To automate this classification, we fine-tune several state-of-the-art large language models (LLMs), including LLaMA, Gemma, Mistral, Qwen, on a manually annotated dataset spanning 10 diverse application domains. Among these, LLaMA 3.1 achieved the highest performance, with weighted F1 scores of 0.90 and 0.79 at the first and second taxonomy levels, respectively. To assess scalability, we applied the fine-tuned LLaMA 3.1 model to label an additional 10,000 unlabeled reviews, creating a silver-standard corpus. This taxonomy-driven approach enables more nuanced analysis of user feedback, supporting targeted app improvements and fostering enhanced user satisfaction.

Large language models (LLMs) are increasingly used for medical consultation and health information support, where safety depends not only on medical knowledge but also on robust responses to unclear, inconsistent, or misleading patient input. However, most existing medical LLM evaluations assume idealized and well-posed patient questions, limiting their realism. We study challenging patient behaviors that commonly arise in real medical consultations and complicate safe clinical reasoning. We define four clinically grounded categories of such behaviors: information contradiction, factual inaccuracy, self-diagnosis, and care resistance. For each behavior, we specify concrete failure criteria that capture unsafe responses. Building on four existing medical dialogue datasets, we introduce CPB-Bench (Challenging Patient Behaviors Benchmark), a bilingual (English and Chinese) benchmark of multi-turn dialogues annotated for these behaviors. We find that although models perform well overall, they exhibit consistent behavior-specific failures, especially when handling contradictory or medically implausible patient information. We further evaluate four intervention strategies and find inconsistent improvements, with some interventions introducing unnecessary corrections. We release the dataset and code.

Predictive vision–language learning aims to learn semantic representations by predicting latent targets rather than reconstructing pixels or generating text. We ask whether latent prediction alone induces the global image–text geometry required for zero-shot matching, and how the location of alignment pressure changes this outcome. In a controlled joint-embedding predictive model family with matched training and evaluation protocols, non-contrastive latent prediction retains information recoverable by supervised adapters and token-level probes but does not form a usable zero-shot image–text space under pooled cosine scoring. A target-contrastive design improves matching, but remains weaker than the directly aligned variant. Our results show that latent prediction is underconstrained at the global embedding level in this setting. Direct global alignment supplies the constraint that makes predictive representations usable for zero-shot scoring. With only a modest direct global alignment loss, latent prediction becomes competitive with matched CLIP and SigLIP baselines and improves several language-sensitive diagnostics.

Vision–language models remain susceptible to multimodal jailbreaks and over-refusal because safety hinges on both visual evidence and user intent, while many alignment pipelines supervise only the final response. To address this, we present SaFeR-ToolKit, which formalizes safety decision-making as an inspectable protocol: during offline data synthesis, a planner specifies a persona, a **Perception**→**Reasoning**→**Decision** tool set, and a constrained transition graph, while a responder emits a typed key–value tool trace before the final answer. To learn this protocol, we train a single policy with a three-stage curriculum (SFT→DPO→GRPO), where GRPO directly supervises tool usage beyond answer-level feedback. We contribute (**I**) the first tool-based safety reasoning dataset of 30,654 training entries (SFT 6k, DPO 18.6k, GRPO 6k) plus 1k held-out evaluation, and (**II**) experiments on Qwen2.5-VL showing large gains in Safety/Helpfulness/Reasoning Rigor on 3B (29.39/45.04/4.98 → 84.40/71.13/78.87) and 7B (53.21/52.92/19.26 → 86.34/80.79/85.34), while preserving general capabilities (3B: 58.67 → 59.21; 7B: 66.39 → 66.81). Code: [https://github.com/Duebassx/SafeR_Toolkit](https://github.com/Duebassx/SafeR_Toolkit).

In this work, we investigate whether detailed, dataset-specific annotation guidelines can replace or augment annotated few-shot exemplars for prompt-based entity detection. To this end, we propose an in-context learning framework that conditions the LLM on a guideline document, task instructions, and optional demonstrations, enabling single-pass, primarily zero-shot nested entity detection. We evaluate multiple instructional aspects from the comprehensive ACE 2005 annotation guidelines, using three guideline-compliant datasets as benchmarks and GPT-5 variants for inference. Based on extensive experimentation, we demonstrate that detailed, dataset-specific instructions, especially those governing mention boundaries, consistently and robustly improve performance, even in conjunction with established few-shot methods. Indeed, even under zero-shot constraints, our guideline approach outperforms state-of-the-art few-shot methods on ACE-2005, achieving up to 81.2 percent F1 without relying on existing labeled datasets. Our findings present detailed annotation guidelines as a promising resource for improving instruction-based information extraction.

Document-level Information Extraction (DocIE) aims to produce an output template with the entities, relations, and events of interest occurring in the given document. Standard practices include prompting decoder-only LLMs using greedy decoding to avoid output variability. Rather than treating this variability as a limitation, we show that sampling can produce substantially better solutions than greedy decoding, especially when using reasoning models. We thus propose ThinkTwice, a sampling and selection framework in which the LLM generates multiple candidate templates for a given document, and a selection module chooses the most suitable one. We introduce both an unsupervised method that exploits agreement across generated outputs, and a supervised selection method using reward models trained on labeled DocIE data. To address the scarcity of golden reasoning trajectories for DocIE, we propose a rejection-sampling-based method to generate silver training data that pairs output templates with reasoning traces. Our experiments show the validity of unsupervised and supervised ThinkTwice, consistently outperforming greedy baselines and the supervised state-of-the-art.

Language models (LMs) are commonly trained with Reinforcement Learning with Verifiable Rewards (RLVR) to enhance their reasoning capabilities. However, since RLVR does not explicitly account for calibration during training, it can lead to severe calibration degradation, including overconfidence. Recent calibration-aware training methods for LMs, which incorporate objectives for uncertainty estimation into training, improve calibration but still exhibit overconfidence under distribution shift, while sacrificing reasoning performance. To this end, we propose RL-ARC, a calibration-aware training framework that jointly leverages reasoning confidence and answer confidence. Specifically, RL-ARC leverages reasoning confidence as an auxiliary signal for calibrating answer confidence, applying it as reasoning-guided regularization for correct cases and as an overconfidence penalty for incorrect cases. Comprehensive results across ID and OOD settings show that, beyond improving calibration, RL-ARC enables reasoning models to adaptively estimate confidence based on the given question without substantially sacrificing reasoning performance, thereby highlighting the importance of reasoning confidence for training reliable reasoning models.

While LLMs demonstrate strong capabilities in English Relation Extraction (RE) tasks, a significant gap remains in assessing their RE performance in low-resource languages, due to a lack of standardized benchmarks. Creating native, language-specific annotated RE benchmarks is expensive and requires human expertise, while relying on translation yields suboptimal, inaccurate datasets that introduce English-centric biases. To tackle this, we propose a scalable, language-agnostic methodology to create RE data by denoising naturally occurring yet noisy alignments of knowledge bases and corresponding texts. We define a proper alignment via a Natural Language Inference (NLI) task, where a relation triplet is entailed by an input text. We empirically assess five denoising strategies and demonstrate that multilingual models outperform language-specific models in the few-shot LLM setting, while Hebrew-specific encoders remain competitive under NLI fine-tuning and are strongest in the label-free cross-training setting. We further find that prompting in the native language achieves performance on par with or slightly exceeding English prompts, confirming that the pipeline operates naturally in the target language. The best open-weights model, Gemma-4-31B-it (F1=0.915), matches the best API model, GPT-5.4 (0.921), and reaches F1=0.844 on a fully held-out, dual-annotated test set. Applying the optimized pipeline, we release HeRE: a novel, massive, 1.5-million-instance silver dataset for Hebrew RE, designed to support data-intensive experimentation and model fine-tuning. HeRE bridges a critical gap in standard Hebrew RE evaluation, while providing the NLP community with a reproducible, language-agnostic pipeline for RE dataset construction in other low-resource settings.

The growing prevalence of multi-sensor wearable devices generates vast amounts of long-term, multimodal time-series data, posing significant challenges for context-aware Large Language Models (LLMs). Current LLM-based health analysis methods typically rely on manually curated context, which becomes impractical with increasing data volume and sensor diversity. To overcome these limitations, we introduce Wearable As Graph (WAG), a novel framework that automates context retrieval for LLMs using personalized knowledge graphs. WAG constructs knowledge graphs mapping relationships between wearable modalities and incorporates user-specific data. We develop a data-driven retrieval pipeline that leverages both global (long-term) and local (short-term) relationships within metrics to identify the most relevant nodes for user queries. We evaluate WAG on a benchmark of over 10k data-associated queries created from multiple wearable datasets. WAG consistently delivers the strongest overall performance, while using substantially fewer input tokens than query-window full-context prompting in the diagnostic setting. These gains remain broadly consistent under human evaluation and across a cross-model generator–judge matrix, supporting structured, query-adaptive context retrieval for LLM-driven analysis of wearable data.

Text-to-SQL is increasingly deployed across trust boundaries between data providers and users. Such deployment must balance three competing requirements: policy compliance, answer coverage, and bounded cost. Existing approaches typically decide refusal based on which columns a query mentions and enforce it stochastically. Whether a query is compliant, however, depends not only on which columns appear but on how they are used, and stochastic enforcement cannot deterministically rule out violations. We formalize this requirement as a column-use policy over semantic use: output, filter condition, and aggregation argument. We integrate the policy by aligning each role with grammar productions tracked by the decoder. The resulting system, PCC-SQL, applies a per-token logits mask that deterministically eliminates single-query column-use violations on the supported SQL fragment in a single decoding pass. Across three benchmarks and three open-source models, PCC-SQL achieves 0% Leakage Rate and Coverage up to 88.7% on Spider-CU, while staying within +10% tokens of direct prompting. We additionally assess semantic alignment with execution accuracy.

In this paper, we survey recent advances in meta-reasoning for Large Reasoning Models (LRMs). Although LRMs have made remarkable progress, they face critical limitations including overthinking on simple queries, under-thinking on complex problems, and ineffective exploration lacking strategic direction. We examine how meta-reasoning addresses these challenges by providing the necessary mechanisms to dynamically regulate and optimize the reasoning process, and propose a framework to categorize existing studies into three core components: the monitor which provides self-perception of the reasoning state, the controller which dictates the meta-policy for regulating actions, and the mechanism which executes the algorithmic mapping between them. By systematically reviewing these developments and foundational challenges, we aim to map the current landscape and identify promising directions for creating more adaptive, efficient, and self-regulating reasoning systems.

PLM-powered neural topic models depend on general-domain pre-training, leading to degraded topic interpretability on specialized corpora. The root cause is geometric: domain-specific terms unseen during pre-training collapse into an indistinguishable region of the embedding space, and neither domain-specific re-training, word-level graph enrichment, nor parameter-efficient fine-tuning can restructure this space without inheriting the capacity ceiling of the underlying encoder. Our key insight is that a learnable graph layer operating on token-level PLM embeddings can acquire corpus-specific semantic structure that the frozen encoder lacks, because token-level graphs preserve document-local context that word-level representations discard and joint optimization with the topic objective reshapes embedding geometry directly from target-domain evidence. We instantiate this insight as DARTopic, a domain-agnostic framework that constructs token-level semantic graphs from frozen PLM embeddings and jointly trains a GNN encoder with variational topic inference. Across three benchmarks spanning general, biomedical, and legal domains, DARTopic consistently outperforms strong baselines in topic coherence and document clustering without any encoder fine-tuning, while demonstrating robustness to PLM choice and favorable runtime efficiency over fine-tuning-based alternatives.

Creative collaboration benefits when collaborators can absorb the nuances embedded in others’ subjective expressions. We focus on especially difficult-to-notice nuances, which we call **minor nuances**, and provide initial empirical evidence that supporting their communication can improve creative collaboration outcomes. We then propose a commonsense-generation-based method that identifies minor nuances in subjective expressions and presents them as cues for collaborators. We evaluated the proposed method in a collaborative image-caption revision experiment designed to reflect brainstorming. Compared with a baseline in which collaborators freely explained the nuances embedded in their own subjective expressions, our method significantly improved output-quality ratings from human evaluators. These results provide initial evidence suggesting that explicitly sharing minor nuances supports richer creative expression by helping convey fine-grained aspects of senders’ inner states that are difficult to communicate through ordinary explanations.

Modeling and predicting political opinions remains a challenging task for AI systems. We introduce PoliSpace, a framework that embeds individuals and political questions into a shared latent space, where vector alignment directly reflects agreement. Individuals are represented as learnable vectors, and questions are encoded via a contrastively fine-tuned language model with individual-specific affine transformations to capture subjective interpretation. This design yields an embedding space that is robust to question rephrasing and sensitive to negation. Evaluated on real-world voting data from Swiss national elections, PoliSpace achieves an 89.81% F1 score on seen opinion prediction for rephrased questions, outperforming prior embedding-based approaches (66.17% F1). We further introduce retrieval and confidence-aware thresholding mechanisms that reach up to 100% F1 at reduced coverage, demonstrating how the learned embedding space can be leveraged to improve prediction reliability. Ablation studies confirm the importance of both individual-specific interpretation modeling and negation-aware encoder fine-tuning. The learned individual embedding space also exhibits clear clustering by political party affiliation, despite no party-label supervision during training, capturing the underlying ideological structure of the Swiss political landscape.

Model merging allows extending an expert model’s capabilities by fusing its weights with another expert. However, task interference has been shown to negatively affect mergeability. To address this, prior work has mainly focused on introducing new merging methods. Despite reducing task interference, the effectiveness of these merging methods is only evident post-merging, requiring a full evaluation of the merged model, which for contemporary LLMs can be expensive. In this work, we study the overlooked predictive power of structural and functional properties of merged experts for mergeability. Our experiments with 8 expert LLMs spanning several programming languages, math, and knowledge show that activation L2 distance and activation cosine similarity emerge as the most important properties for predicting effective merging, contrasting with prior findings on vision models where gradient-based metrics dominated. We further show that task vector L2 distance, a cheaply computed structural metric, provides a mechanistic link between mergeability prediction and training dynamics: it explains why learning rate effects on mergeability are domain-dependent in LLMs, contradicting the universal “higher noise improves merging” principle from vision. This enables a practical intervention where mergeability metrics guide learning rate selection during expert training.

Retrieval-Augmented Generation (RAG) mitigates key limitations of Large Language Models (LLMs), such as factual errors, outdated knowledge, and hallucinations, by dynamically retrieving external information. Recent work extends this paradigm through agentic RAG systems, where LLMs act as agents to iteratively plan, retrieve, and reason over complex queries. However, these systems still struggle with challenging multi-hop questions, and their intermediate reasoning capabilities remain underexplored. To address this, we propose RAGCap-Bench, a capability-oriented benchmark for fine-grained evaluation of intermediate tasks in agentic RAG workflows. We analyze outputs from state-of-the-art systems to identify common tasks and the core capabilities required for their execution, then construct a taxonomy of typical LLM errors to design targeted evaluation questions. Experiments show that “slow-thinking” models with stronger RAGCap performance achieve better end-to-end results, underscoring the benchmark’s validity and the importance of enhancing these intermediate capabilities.

Long-term memory significantly enhances language-model agents but introduces a critical failure mode: retrieval surfaces relevant yet stale, deleted, or mutually inconsistent facts. A query about a current address may retrieve the correct address alongside deleted ones. The answer is present, but the memory set is unsafe. We reframe agent memory as a governed object-selection problem and present MemGov, a substrate-agnostic middleware layer between existing retrieval pipelines and agent context. MemGov wraps retrieved evidence as explicit memory objects, detects must-link and cannot-link constraints, and applies deterministic constraint-aware repair, returning either a certified safe memory set or an auditable obstruction certificate rather than silently passing unsafe combinations. Evaluations on controlled governance scenarios and MemoryAgentBench-style end-to-end tasks, deliberately separating retrieval quality from governance decisions, demonstrate that while raw retrieval recovers answer-bearing evidence successfully, it simultaneously returns detected unsafe memory combinations in 47-51% of cases. The same retrieval substrates governed by MemGov reduce unsafe merges to 0% while preserving answer utility within about 2-4 percentage points, with fully auditable reasoning for all repair and refusal decisions. Additional diagnostics assess eligibility, temporal constraints, and paired answer retention.

Text-driven indoor scene generation and editing require an intermediate representation that language models can both produce and revise. Existing LLM-based systems often rely on scene graphs or global constraint lists, which are compact but underspecify local geometry and make instruction-based edits difficult to localize. We frame the task as structured program generation and local program repair, and propose the Hierarchical Descriptive Scene Language (HDSL), an XML/CSS-style domain-specific language in which rooms, regions, objects, and support surfaces form a tree with parent-relative local coordinates. Our pipeline generates HDSL subtrees recursively with bounded verification, grounds non-virtual nodes through multimodal asset retrieval, and applies force-directed layout optimization to repair boundary and collision errors. For editing, Hierarchical Retrieval-Augmented Generation retrieves the relevant subtree, rewrites only that local context, and merges the result back through a deterministic three-way merge. In our reproduced benchmark, HDSL improves average object coverage, text-scene alignment, and generation time over full text-to-scene baselines while remaining competitive with layout-only reproductions on geometry metrics. On thirty paired editing instructions, HRAG reduces token use by 67.6% and runtime by 68.1%, and cuts non-target removals by 87.5% over full-scene rewriting; attribute ablations confirm that local coordinates and virtual/leaf flags are necessary for valid, locality-preserving edits.

Autoregressive language models are usually trained with next-token prediction under a fixed left-to-right target order. This paper studies target-token organization as a training-time design choice across arithmetic reasoning, multi-label text classification, and text generation. Controlled order studies show that the default NTP order can be far from the best-performing tested organization, while fixed hand-crafted orders such as reverse order do not generalize across tasks. To address this, Max(MI(S; t)) is introduced as a deterministic mutual-information-based heuristic that prioritizes information-rich target tokens. For fixed-length outputs, it constructs task-specific prediction orders; for open-vocabulary text generation, the same principle is applied through lightweight prefix augmentation. Experiments with GPT-style models and pretrained Llama/Qwen variants show that Max(MI(S; t)) often improves over standard NTP and simple baselines such as reverse order, TF-IDF, and conditional-probability selection, without changing model architecture or autoregressive decoding.

LLM-as-a-Judge has emerged as a scalable alternative to human evaluation in multilingual cultural reasoning, yet its reliability for culturally grounded judgments remains poorly understood. We present a human-aligned validation study using the CRaFT framework across Arabic, Bengali, and Spanish, evaluating six judge models, including five open-weight models and GPT-4.1, against paired human annotations across two structurally observable dimensions (Answer Consistency and Explanation Consistency) and three culturally interpretive dimensions (Cultural Fluency, Linguistic Adaptation, and Deviation). Our results reveal a consistent reliability divide: automated judges align substantially better with humans on structural dimensions than on culturally interpretive ones requiring sociocultural reasoning. Reliability further varies across languages, judge architectures, and response-generation models, with several failures reflecting systematic calibration limits rather than prompt underspecification alone. These findings suggest that current automated evaluators are best treated as complementary diagnostic tools rather than replacements for human judgment in culturally sensitive multilingual settings.

Large language models (LLMs) are increasingly studied for clinical decision support, and medical question answering provides an effective testbed for evaluating this capability. In particular, medical multiple-choice questions (MCQs) reflect a core challenge in clinical decision-making: plausible options often share substantial overlapping evidence, while only a few decisive factors distinguish the correct option from other incorrect ones. However, existing retrieval-based and reasoning-oriented methods mainly focus on supplying broadly relevant knowledge or eliciting reasoning paths, rather than identifying the compact differential evidence that separates confusing alternatives. In this work, we present a retrieval-grounded differential evidence framework for medical MCQs. Our method extracts option-specific pieces of differential evidence and organizes them into concise evidence prompts for answer selection. This design better aligns external knowledge with differential reasoning while reducing irrelevant context. Experiments show consistent accuracy gains over strong baselines. Ablation studies further show that our framework outperforms both directly using retrieved evidence chunks and closed-source LLM-generated general hints, and that its improvements are not solely explained by answer-proximal guidance, but are consistent with differential evidence serving as a compact aid for reasoning over competing alternatives. Our findings also offer insights for medical reasoning and other decision-making settings with multiple possible choices.

The rapid proliferation of Vision-Language Models (VLMs) is often framed as enabling unified multimodal understanding, but rests on an under-examined assumption: that current VLMs faithfully synthesise multimodal data. We argue they often do not, and that this gap reflects a trustworthiness problem in the dominant Vision Encoder-Projector-LLM paradigm. Rather than extracting grounded knowledge from visual inputs, state-of-the-art models frequently exhibit functional blindness, exploiting strong language priors to bypass severe visual representation bottlenecks. We challenge the conventional methodology of multimodal evaluation, which relies on data ablation or new dataset creation and therefore conflates dataset biases with architectural incapacity. We propose an information-theoretic departure: the *Modality Translation Protocol*, which preserves a sample’s semantic payload while translating its modality rather than ablating it. From this, we derive three metrics—the Toll (*ToS*), Curse (*CoS*), and Fallacy (*FoS*) of Seeing—culminating in the *Semantic Sufficiency Criterion (SSC)*, a single condition that detects, quantifies, and localises multimodal failures to the visual encoder or the projection head. We further hypothesise a *Divergence Law of Multimodal Scaling*: as language backbones grow, the penalty of the visual bottleneck may widen rather than close. Examining the closest-matching existing benchmarks, motivates our central call: the construction, through *Semantic Equivalence Engineering*, of a large, diverse dataset whose modalities are isomorphic by construction. We frame this not as an experiment awaiting compute, but as a mission for the community: to move beyond "multimodal gain" and build the substrate on which faithful multimodal architectures can be designed.

The rapid expansion of scientific literature far outpaces the speed of manual curation, creating a fundamental bottleneck in extracting structured knowledge from scientific texts. To address this, we introduce an agentic, ontology-guided framework for knowledge graph (KG) construction. Our framework utilizes specialized agents for document understanding, information extraction, ontology conformance, and entity linking. First, an LLM extracts factual knowledge in the form of triples with explicit textual evidence from research papers. Then the triplets undergo a multistage verification process to ground the triplets in the ontology. By enforcing schema constraints and requiring evidence in the form of text fragments, we ensure the data is both semantically consistent and traceable. In an agreement study over 1,000 extracted triplets, Gemini-2.5-flash achieves Cohen’s 𝜅 scores of 0.640–0.700 against three human annotators, matching the human–human agreement range of 0.616–0.689. We applied this framework to 2,000 scientific publications and construct a novel, ontology-grounded KG for climate science with 133,990 nodes and 755,308 relations; of the 23,327 triplets that the publications contribute after validation, 98.96% are judged correct. The pipeline code and KG is publicly available.

A central question in LLM reasoning is whether reinforcement learning (RL) instills genuinely new capabilities or merely reshapes how existing knowledge is expressed during inference. Building on the distribution-sharpening hypothesis, which holds that RL reallocates probability mass toward high-reward trajectories already latent in base models, we ask: can we unlock those latent paths without costly RL fine-tuning? We present Decision-Flow Sampling (DF-Sample), a training-free, data-free inference-time framework that constructs a hierarchical reasoning tree, scores terminal nodes for quality, and back-propagates utilities to inform each intermediate branching decision. Unlike conventional sampling strategies that make purely local step-wise choices, DF-Sample performs explicit global trajectory evaluation before committing to a path, recovering high-quality but low-probability reasoning chains that standard decoding overlooks. On GPQA, DF-Sample achieves 45.6% accuracy, surpassing power sampling (38.9%) and GRPO (39.9%), showing that a training-free method can outperform a trained one. Across three models and four benchmarks, DF-Sample consistently outperforms baselines, indicating substantial latent reasoning potential in pretrained base models.

Large language models are increasingly used for Legal Judgment Prediction (LJP), raising concerns about demographic bias inherited from pre-training corpora and historical legal records. We investigate how fairness robustness changes under model adaptation in LJP through a controlled counterfactual evaluation across gender, ethnicity, and intersectional demographic groups. Using 23 demographic variants over encoder-based models and generative LLMs, we evaluate demographic sensitivity, counterfactual instability, and bias propagation under zero-shot, few-shot, and fine-tuned settings. Experiments show that model adaptation substantially affects fairness. Fine-tuning often improves predictive performance while simultaneously increasing demographic sensitivity and propagating biased supervision signals into downstream judicial predictions, whereas few-shot prompting generally produces smaller propagation effects. We further show that legal-domain specialization does not reliably guarantee fairness: legal-domain encoders span both highly robust and highly sensitive behavior after fine-tuning, and a legal-domain LLM is highly sensitive under few-shot prompting yet comparatively robust after fine-tuning under counterfactual perturbation.

Parameter-Efficient Fine-Tuning (PEFT) has become the standard approach for adapting large language models (LLMs). However, despite reducing trainable parameter counts, many PEFT methods remain memory-bound by token-level activation intermediates that scale linearly with sequence length, limiting their practicality for long-context and on-device adaptation. Unlike prior memory-efficient PEFT approaches that preserve token-level adapter activations, LARS (Low-memory Activation-Rank Subspace) reduces the sequence-length scaling of stored adapter activation intermediates from O(BSRL) to O(BRL) by performing adaptation within a sequence-level low-rank subspace. Through pooled feature extraction, low-rank subspace modulations, and residual projection, LARS reduces the overall memory footprint by 23% on GPUs and 47.8% on CPUs relative to LoRA at S = 1024, with relative savings increasing with context length. Experiments across reasoning, understanding, long-context, retrieval, and token-level tasks show that LARS maintains competitive performance relative to LoRA while substantially improving memory efficiency. Beyond GPUs, LARS also reduces average CPU power and energy consumption per optimization step by 22.43% and 22.06%, respectively. These results demonstrate that activation-efficient adaptation provides a practical path toward resource-constrained and on-device fine-tuning of LLMs.

Large language models (LLMs) often answer ambiguous inputs as if they had a single intended meaning. This is problematic because input ambiguity is an upstream source of uncertainty: when a prompt admits multiple valid interpretations, the key question is whether the model should answer directly at all. Existing input uncertainty methods often rely on LLM-generated statements or clarifications, but such transformations may already disambiguate the input before the ambiguity is measured. We propose a label-preserving perturbation framework for input ambiguity detection in which perturbations remain close to the original input while preserving its label. We instantiate this principle with two detectors: a black-box rule-based perturbation library and a white-box importance-guided method using model-internal signals. Experiments across four datasets in open-domain, conversational, and visual QA show that our methods outperform output-side uncertainty baselines and LLM-paraphrase-based perturbation baselines. Our results highlight the importance of detecting ambiguity before attempting to disambiguate it. Our code is available here.

Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for improving the reasoning capabilities of LLMs. However, the role of supervised fine-tuning (SFT) data in enabling effective downstream RL remains underexplored. In this paper, we investigate a key hypothesis: the diversity of reasoning steps in chain-of-thought (CoT) during SFT is a critical enabler of RLVR performance. To test this hypothesis, we introduce EquivAug, a data augmentation framework that uses symbolic equivalence transformations to generate diverse yet provably correct reasoning paths. By converting CoT solutions into formal symbolic representations, applying equivalence transformations, and reconstructing natural-language explanations, EquivAug produces training data that differs in reasoning steps rather than surface-level lexical variation. Experiments on GSM8K and FOLIO across model scales demonstrate that EquivAug consistently achieves the highest reasoning path diversity after SFT and the strongest accuracy after GRPO. Our results establish that CoT diversity during SFT is positively correlated with downstream RLVR performance.

Sparse autoencoder (SAE)-based activation steering is a lightweight approach for modifying the behavior of large language models (LLMs) by adding small vectors to hidden states during inference. However, designing effective steering interventions remains challenging. In this work, we develop an information-theoretic framework for analyzing SAE activations through empirical feature-firing distributions. The central object of the framework is the Jensen–Shannon divergence (JSD) between such firing distributions, which quantifies how far a generated or intervened state deviates from a dataset-induced target distribution. Based on an analytic divergence response for virtually shifted states, we construct steering vectors toward the target firing distribution. The same divergence can also be used to monitor deviation from another empirical distribution, such as a general valid-language distribution. We demonstrate the effectiveness of the steering in a question-mode generation task study. We further use teacher-forced pulse experiments to show that local interventions induce structured downstream JSD responses in later layers, and apply the framework to a safety-related target–contrast response setting for feature analysis.

Visualized text retrieval, which renders text into images and retrieves them with vision-language models, has been proposed to avoid lossy text parsing and compress long contexts. We ask whether it also bypasses a known flaw of text retrieval: position bias, the tendency to overweight a passage’s beginning. Rendering 1D text into a 2D image may change the bias in two respects: its shape, which position is favored, and its magnitude, how strongly performance varies across positions. We test this by rendering the PosIR benchmark as controlled plain text images and comparing eight visual retrievers with four text retrievers on identical instances. We find that (1) position bias persists, with all visual retrievers exhibiting measurable bias (macro PSI 0.061–0.168); (2) its shape is transformed: seven of eight visual retrievers favor the middle of the page in an inverted-U pattern, contrasting with both the primacy bias of text retrievers and the lost-in-the-middle effect of LLMs with long contexts; (3) its magnitude is sensitive to rendering settings, as taller images amplify PSI and can even reverse positional preference, while font size has a similarly large effect; and (4) under matched backbones, visual retrievers exhibit lower position bias than their text counterparts on longer documents, but this advantage vanishes beyond the pixel budget and at document scale. Visualized text retrieval therefore does not eliminate position bias; it transforms a familiar 1D textual bias into a more complex 2D spatial bias.

Large Language Models (LLMs) often struggle to navigate user requests that contain complex value conflicts, as direct scalar rewards are often ill-equipped to handle these scenarios. To address this challenge, we investigate how chain-of-thought (CoT) reasoning can help improve performance in this domain. Geometrically, we show that CoT correlates with further smoothing the model’s loss landscape in its sharpest direction, helping resolve the optimization instability of traditional scalar rewards. In terms of performance, we demonstrate via relevant downstream benchmarks that CoT may increase capability, demonstrating that value conflict-focused CoT has the potential to be an effective mechanism for better moral reasoning. To capitalize on this potential, we create a new value conflict-focused CoT design that correlates with further smoothing the sharpest direction of the loss landscape and further increases moral reasoning performance. This finding shows that explicitly modifying and improving the design of reasoning dynamics offers a promising avenue for advancing pluralistic alignment in LLMs.

Large language models (LLMs) demonstrate strong instruction-following and reasoning abilities, yet struggle with the socially nuanced understanding humans handle naturally. Humans rely on social sense to interpret intentions, emotions, and relationships in context. We investigate whether and to what extent LLMs exhibit similar abilities when reasoning about social interactions. We propose a structured approach that decomposes social interactions into three aspects and prompts LLMs to generate aspect-based summaries capturing complementary signals. We evaluate these summaries and resulting reasoning within a unified framework across three dimensions, analyzing both LLM–human alignment and the quality of LLM-generated social explanations. Our results show that LLMs achieve moderate agreement with humans and with one another as evaluators and capture some socially nuanced information across aspects, but struggle with interaction-level dynamics shaped by situational norms. Across four downstream CSS tasks, aspect summaries improve aggregate prediction performance across three model families, though effects vary by task. Complementary rationale analysis shows that summaries substantially alter expressed reasoning. Finally, we introduce a benchmark dataset with gold-standard annotations to support systematic study of social understanding and situated reasoning in LLMs.

Large Audio Language Models (LALMs) exhibit strong speech understanding ability, yet it remains unclear whether they meaningfully utilize subtle speech-conditioned emotional cues beyond coarse acted emotions. In this work, we investigate whether the evaluated LALMs exhibit measurable sensitivity to fine-grained emotional variation in speech through controlled perturbation experiments, fine-grained emotion elicitation tasks, and representation geometry analysis. Our results show that vocal delivery variations systematically influence both model behavior and hidden representations, indicating that the evaluated LALMs already encode non-trivial speech-conditioned affective information. However, models struggle under emotionally ambiguous settings, particularly for semantically related fine-grained emotions. Representation analysis further suggests that subtle emotional structure is partially preserved despite weak downstream utilization. Overall, our findings suggest that the evaluated LALMs do not completely ignore subtle speech emotion cues, but instead utilize them inconsistently under ambiguous affective conditions

Policy gradient methods are widely used to align large language model outputs with human preferences, but the log-likelihoods they require cannot be directly computed in Masked Diffusion Language Models (MDLMs). MDLM post-training therefore relies on a surrogate objective based on the Evidence Lower Bound (ELBO), and recent work estimates a blockwise variant of this objective via Monte Carlo methods to support blockwise generation. However, existing estimation schemes randomly sample the block index, leading to unstable estimation. In this work, we propose a full-block estimator that evaluates every block exactly once and averages the results. We theoretically show that the proposed estimator remains unbiased while eliminating the between-group variance arising from block selection, thereby always achieving lower variance than the baseline for the same sample size. In experiments on five benchmarks spanning mathematics, planning, and coding, our method achieves the highest accuracy on four benchmarks and outperforms the random block sampling baseline by an average of 12.79 points.

Language model (LM)-powered agents have shown promise in addressing complex embodied tasks by leveraging the commonsense world knowledge of pre-trained models in interactive physical environments. Yet, real-world deployment requires agents to continually adapt to a stream of tasks, where neither large-scale expert supervision nor reliable external feedback is consistently available. In this paper, we present ASC, a framework for efficient continual embodied task adaptation through amortizing self-correction. Specifically, ASC trains the policy under two objectives: a standard CoT reasoning objective distilled from stronger experts, and a reflexive reasoning objective that internalizes self-correction by correcting initial rationales generated by a previously learned policy. This dual optimization allows the agent to adapt more efficiently through forward knowledge transfer over time, ultimately fostering robust CoT reasoning at inference without requiring explicit reflection. Furthermore, we introduce rationale-wise test-time scaling that dynamically adjusts the depth of CoT reasoning based on the model’s confidence in actions inferred from its own rationales. Through experiments on VirtualHome and ALFWorld, we demonstrate performance superiority over other LM-based planning and continual task adaptation approaches, while achieving strong efficiency in computation, data usage.

Machine-generated text detection remains difficult in Chinese, where systematic benchmarks and large paired corpora are still scarce. We revisit this problem with a compact 4-layer character-level Transformer, trained on a 363.4-million-character Chinese detection corpus, to our knowledge the largest AI-human paired Chinese resource for this task. The resulting data-centric pipeline attains strong scores on Chinese benchmarks and, for the independently trained English variant, state-of-the-art RAID leaderboard numbers that remain stable under 11 RAID adversarial attack types. Ablations expose a double-edged sword role for punctuation: it eases optimization yet encourages overfitting to surface cues; symbol cleaning shifts the model toward deeper syntactic and discourse patterns and improves generalization. Minimal architecture plus rich supervision thus yields compact detectors suitable as a document-level authorship authenticity screen before downstream Chinese text processing; on a laptop RTX 3070 Ti we measure 5.90 x 10^5 tokens/s at under 100 MB VRAM. We present GiantMelon-ZH (Chinese) and GiantMelon-EN (English for RAID): same backbone, disjoint training corpora, transparent data descriptions.

Simulated-patient benchmarks are widely used to evaluate clinical AI agents, yet the patient simulator itself can alter the evidence available to the agent and therefore the conclusions drawn from the benchmark. We introduce PULSE-CARE, a measurement-validity audit for simulated-patient clinical-agent evaluation that quantifies sensitivity to patient simulator behavior, agent architecture, simulator backbone, scorer choice, and demographic framing. Across 24 synthetic longitudinal cases, four dynamic simulator variants, three agent models, and two architectures, the mean per-case score range across simulators is 0.25 (95% CI: 0.21-0.29), and escalation decisions differ across simulator conditions for 60% of agent-case pairs. A 392-trajectory repeated-sampling analysis shows that simulator effects persist after pooling repeated draws, while also revealing substantial within-policy stochastic variation. Nominal model rank reversals occur in 17% of comparisons but disappear under a 0.05 practical-difference margin. Demographic counterfactuals change escalation in 26% of matched pairs, essentially the same rate as the same-input repeated-sampling control, providing no evidence of a demographic effect in this audit. PULSE-CARE combines simulator-sensitivity metrics, EFI composite indices, Patient Simulator Cards, and an Audit Report template to make the measurement conditions behind simulated-patient benchmark results explicit. Our results show that clinical-agent performance reported from a single patient simulator should be interpreted as conditional on the simulator and evaluation pipeline used to obtain it.

Supervised fine-tuning (SFT) is typically viewed as behavior cloning on expert demonstrations. Recent methods recover richer supervision from demonstrations through self-play preference synthesis or bilevel inverse reinforcement learning, yet these require additional optimization stages or separately trained reward models. Motivated by the connection between SFT and inverse soft-Q learning, we propose Dense-Path REINFORCE (DPR), a lightweight post-SFT policy-improvement method that extracts a dense reward signal directly from the SFT training trajectory. We construct a token-level reward from the log-probability gap between the final SFT policy and an intermediate checkpoint, capturing the incremental likelihood gain acquired during training while mitigating the length and early-EOS biases inherent in raw log-probability rewards. DPR reuses the same demonstrations and prompts as SFT, requires no preference data, no external teacher, and no separately trained reward model. Across four pretrained backbones and four instruction-following evaluations, DPR obtains 55.2-66.8% win rates over SFT, improves MT-Bench by 0.22-0.46 points, and matches or exceeds the best demonstration-only baseline in 15 of 16 settings.

Chain-of-thought traces drive strong LLM accuracy but dominate inference cost: output length sets both monetary cost and decoding latency. We ask which tokens in a reasoning trace actually carry the answer. A token-level analysis of the model’s own next-token distribution identifies three redundancies - discourse-element removal, tokenizer-aware abbreviation, and substitution of prose causal connectives with an arrow "X -> Y". All three leave the predictive distribution close to the original under Jensen-Shannon divergence and preserve the greedy next token in the vast majority of prefixes. We instantiate the analysis as CONCISON, a single training-free system prompt that operationalises the three rules and protects numeric, equation, and code tokens verbatim. Across 18 models from six families (2B-120B) and five reasoning benchmarks, CONCISON cuts output tokens by 24-62% on strong models at -1.1pp average accuracy, outperforms token-budget prompting in a single decoding call, and matches trained routers on the accuracy-cost frontier without any training, model pool, or extra infrastructure.

Machine unlearning aims to remove the influence of specific data from trained language models. In real-world deployments, unlearning requests often arrive sequentially, which challenges existing fine-tuning-based methods: fine-tuning each request is costly, accumulates utility loss, and may cause cross-request interference. To address these issues, we propose ICCU (In-Context Continual Unlearning), an in-context continual unlearning framework that induces readable refusal rules from unlearning datasets and applies them at inference time either as a filter or via the system prompt, without modifying model parameters. Because rules are accumulated as an order-independent union, ICCU is compositional and free of cross-request interference, and the original forget-set data can be discarded after rule induction. Extensive experiments show that ICCU effectively suppresses target knowledge while preserving utility, scales across long sequences of requests, and remains robust to paraphrased, cross-lingual, and adversarially framed queries. Our code is available at https://github.com/rhpan/ICCU.

Since no metrics are available to evaluate specific aspects of a text, such as its personalization quality or quality of adaptation to a specific form, researchers often rely solely on large language models to evaluate such texts. Due to internal biases of individual language models, it is recommended to use multiple of them for combined evaluation, which directly increases costs of such evaluation. In this paper, a computationally efficient method (called PerQ) for evaluation of personalization quality (in terms of platform adaptation) of a given text (generated by a language model) is introduced, which is based on predicting the LLM judgments (i.e., a kind of metric distillation). A case study of comparison of generation capabilities of large and small language models shows the usability of the proposed metric in research, effectively reducing the waste of resources.

Reliable cross-modal reasoning requires Multimodal Large Language Model (MLLMs) with interpretable and controllable computation. However, existing models rely on data-driven architectures that entangle multimodal processing, obscuring the functional processes underlying their outputs. Inspired by the human brain’s specialized and dynamically collaborative networks, we propose **BrainMoE**, a multimodal MoE model with six sensory and cognitive experts: Language, Logic, Social, World Knowledge, Vision, and Auditory. To support flexible sensory–cognitive coordination, we incorporates a layer-wise adaptive router that dynamically determines which experts are activated at each layer based on task-processing demands. Experiments on text, image, and audio reasoning benchmarks demonstrate that BrainMoE outperforms strong baselines by 1.59 and 2.2 points on text-only and multimodal benchmarks, while matching competitive external multimodal models. Routing and ablation analyses demonstrate clear expert specialization and causally important module-level contributions. Experiments on the cognitive benchmark further show that adaptive-routing BrainMoE aligns more closely with human behavioral patterns than baselines and static routing methods.

Recent advances in large language models(LLMs) have given rise to large reasoning models that achieve strong performance through explicit multi-step reasoning. However, models often generate long reasoning chains even for simple inputs. We present Difficulty-Aware Long-to-Short LLM Reasoning with Model Merging (DALSA), an inference-time framework that adaptively controls reasoning depth based on input difficulty. DALSA integrates (i) representation-based difficulty estimation, which predicts a soft failure-rate score from the prompt hidden state, and (ii) difficulty-aware model merging, which uses a quantile remapped score to modulate the interpolation coefficient between short- and long-reasoning models. DALSA trains a lightweight probe while keeping both LLMs frozen. On mathematical benchmarks with the evaluated Qwen pairs, the accuracy–length trade-off depends on scale: at 7B, DALSA preserves static ACM-TA average accuracy (66.60% vs. 66.61%) while reducing average response length by 22.35%; at 1.5B, it improves average accuracy from 56.75% to 58.24% with a 2.27% length in- crease. Additional merging baselines, routing-threshold and merge-range sweeps, and component ablations show sensitivity to the operating configuration. Domain-matched commonsense and Llama-family diagnostics do not establish broader efficiency gains, limiting the primary empirical claim to the evaluated Qwen mathematical setting.

Multi-trait automated essay scoring (AES) provides trait-level scores as feedback but requires costly annotations across multiple traits to train accurate scoring models. To alleviate this annotation burden, we propose MTPAS (Multi-Trait Preference-to-Absolute Scoring), a semi-supervised framework that learns relative score orderings from LLM-generated pairwise preferences and calibrates them to absolute rubric scales using a small set of human-scored anchor essays in an end-to-end manner. To this end, MTPAS jointly optimizes three objectives: a Bradley–Terry pairwise preference loss, a regression loss on the anchor essays, and a Sinkhorn divergence-based distribution regularizer. The model adopts a bifactor architecture that decomposes essay representations into a shared general factor and trait-specific residuals, improving robustness against noisy LLM-generated preferences. We further introduce pairwise rubric compression, which reformulates the original score-level rubrics into concise comparative criteria, reducing the inference cost of collecting LLM preferences. On ASAP and ASAP++, MTPAS achieves 91.4% of the fully supervised SoTA performance with only 30 anchor essays per prompt and further narrows the gap with 200 anchors. Under the same anchor budget, MTPAS outperforms conventional multi-trait AES baselines at every anchor size investigated in this study.

In conversational and cinematic subtitle translation, large language models often exhibit stylistic collapse, defaulting to neutral registers that flatten character-specific personas. Prompt-based attempts to recover style frequently introduce instruction drift and semantic degradation. We propose Orthogonal Style Injection (OSI), a parameter-efficient, inference-time intervention that removes global stylistic bias through orthogonal projection at an intermediate layer and injects a character-specific style vector derived solely from English source dialogue. We evaluate OSI for English-to-Bengali, Hindi, Telugu, Tamil, Kannada, Chinese, Japanese, Spanish, French, German, and Italian subtitle translation, spanning 11 typologically diverse target languages. Results show that zero-shot stylistic transfer depends strongly on pretraining alignment. Llama-3.1-8B yields gains in several European languages but degrades on Indic languages, whereas Qwen-2.5-7B transfers style more robustly across Indic and East Asian languages. To probe sociolinguistic mechanisms, we curate a parallel English-Bengali cinematic dataset as a focused morphological stress test. Automatic metrics and human evaluation show that OSI enables meaningful honorific and register shifts while largely preserving semantic fidelity, revealing a dataset-dependent trade-off between stylistic fidelity and translation quality in highly colloquial, morphologically constrained settings.