Pragmatic understanding remains a fundamental challenge in natural language processing, particularly for underrepresented languages such as Cantonese. To alleviate the scarcity of resources in this area, we introduce YUE-PUB, the first large-scale pragmatic understanding benchmark for Cantonese. Constructed from the PUB dataset through automatic translation with human verification for quality assurance, YUE-PUB covers three core pragmatic phenomena: Implicature, Presupposition, and Reference. We conduct systematic experiments across multiple baseline models to evaluate their performance under several cross-lingual transfer settings, compare models with and without pragmatic topic awareness, and further analyze their sensitivity to different pragmatic phenomena as well as their Chain-of-Thought (CoT) reasoning behavior. Experimental results show that current large language models still exhibit substantial limitations in Cantonese pragmatic understanding, underscoring the value of YUE-PUB as an important resource for advancing pragmatic reasoning research in low-resource languages.

Recent studies have shown that code-switching data (CSD), in which multiple languages are mixed within the same context, can improve cross-lingual transfer and multilingual alignment in large language models (LLMs). Recent studies on synthetic code-switching for LLMs have primarily focused on bilingual transfer between English and a target language, leaving instruction-tuning settings that jointly mix three or more languages within each example largely unexplored. In this work, we investigate multilingual code-switching instruction tuning across four languages: English, Japanese, Korean, and Chinese. We evaluate multilingual understanding on Belebele and MMLU. Our experiments show that simple sentence-level multilingual CSD consistently improves the four-language average accuracy across all bilingual, trilingual, and quadrilingual settings for Gemma-2-2B, indicating that multilingual code-switching can be effective beyond bilingual transfer settings.

Corpus Query Language (CQL) supports precise corpus analysis but requires expertise in annotations, regular expressions, structural scopes, and platform-specific syntax. Natural-language interfaces simplify access, but their answers can be difficult to inspect, reproduce, or revise. We study Text-to-CQL as executable semantic parsing, mapping intents to inspectable queries. We introduce ECQL, a Sketch-Engine-oriented benchmark of 3,572 natural language–CQL pairs comprising 811 tutorial-derived and 2,761 synthetic examples, with a default split of 3,215 training and 357 test examples. ECQL focuses on Sketch Engine normalization, CQL Diagnostic Profile (CQL-DP) annotations, controlled semantic contrasts, audit-oriented metadata, and generalization protocols, alongside an evaluation suite covering syntactic validity, exact match, canonical match, and execution match. The full 357-example headline test set is independently double-annotated and adjudicated, with 96.9% agreement for intent clarity, 94.7% agreement for CQL–intent alignment, and a 3.1% post-adjudication intent-match error rate. Experiments with prompting, retrieval-augmented prompting, syntax repair, DeepSeek-R1 reasoning prompting, a GPT-4o Agent/API alternative, and supervised fine-tuning reveal persistent difficulties with corpus-query semantics. The best fine-tuned model reaches 85.6% execution accuracy on the random split, but drops to 68.9% on template-disjoint evaluation. ECQL provides a reproducible testbed for auditable semantic parsing over specialized linguistic query languages.

Although large language models (LLMs) show promise for generating educational feedback, it remains unclear how feedback should be designed (e.g., tone and coverage) to support answer revision and learner evaluations across learner profiles. We define six feedback designs for multiple-choice biology questions, including a baseline design and five variants with additional feedback elements, and conduct an empirical study with 321 high school students. We evaluate feedback using immediate revision performance and six subjective evaluation criteria, and analyze differences in subjective evaluations across learner profiles based on personality traits. Our results show that presenting task-relevant information clearly is associated with better immediate revision performance and is favorably evaluated across learner profiles, while we observe descriptive differences in evaluation patterns, particularly for informational novelty and affective framing. These findings support further investigation of personalized LLM feedback design.

We introduce Extended-SVD Low-Rank Adaptation, a parameter-efficient framework that extends the pretrained singular subspace while retaining a frozen spectral coordinate system and fixed singular-value-derived scaling. Its stabilized variant, PLORA (Procrustes Low-Rank Adaptation), applies localized Procrustes-inspired reorientation to the extended basis and confines task-specific learning to compact routing matrices P, R, and Q.

Across GLUE and vision benchmarks, PLORA achieves competitive performance with substantially fewer trainable parameters. In the full-network comparison, PLORA-r8 uses substantially fewer trainable parameters, although the performance gap varies by task. Beyond accuracy, PLORA produces task- and architecture-dependent changes in conditioning, effective rank, and subspace evolution.

Ablations show that Random Bases produce poorer routing-core conditioning and markedly lower top-r right-subspace overlap in the evaluated NLP tasks. In the evaluated settings, pretrained spectral bases are associated with more favorable optimization geometry, while Procrustes reorientation acts as an optimization-geometric regularizer. Overall, Extended-SVD provides a structured low-rank adaptation framework for studying the role of pretrained geometry in PEFT.

Retrieval-Augmented Generation (RAG) systems are increasingly used for biomedical Question Answering (QA). However, evaluating their effectiveness remains challenging due to the lack of reliable reference resources and the frequent misalignment between automatic evaluation metrics and human judgments. One practical question is how evaluation results should be used to guide development decisions, especially when they correlate weakly with human judgments and may even lead to different conclusions. We explore how to use a small amount of human preference signals to calibrate direct assessment evaluation metrics, and show that calibrated metrics consistently outperform baselines that either treat all metrics equally or rely on a single best metric across multiple datasets.

Large Language Models often generate factually correct responses that fail to maintain user-requested perspectives, blending multiple perspectives and forcing users to filter through unwanted information—a perspective control failure that factual accuracy metrics cannot detect. We introduce PerspectiveBench, a benchmark quantifying perspective alignment by constructing scenarios where multiple valid reasoning paths coexist, and propose the Perspective Alignment Score (PAS) to measure adherence to specified perspectives. Experiments reveal that state-of-the-art models successfully include target content but fail to exclude non-requested perspectives. Providing documents through RAG improves factual grounding but does not resolve perspective drift, as models prioritize comprehensive coverage over strict user constraints. Code and data are available at https://github.com/sunghoon014/PerspectiveBench.

Post-training quantization methods such as GPTQ and AWQ minimize reconstruction error without asking which weights matter for the task. We combine six interpretability signals to select a sparse skeleton of task-critical MLP neurons that is kept at 16-bit while the rest of the model is quantized. On Gemma-2-2B fine-tuned for SQL generation, the 0.54% skeleton with 3-bit MLPs and 8-bit attention retains 100% of baseline-correct outputs at 4.03x compression, versus 75.2% for uniform 4-bit. On Llama-3-8B fine-tuned for Cypher generation, where uniform 4-bit loses 23 points, the 0.46% skeleton recovers 14.4 of them and a random skeleton of equal size recovers none. Three signals are individually effective and three are not, and the effective set differs between models, so the consensus rule provides robustness no single signal does. Selection matters where uniform quantization leaves headroom, which suggests measuring headroom, selecting by consensus, and protecting as a recipe for larger models. GPTQ error compensation degrades 3-bit layouts by 16 to 18 points regardless of protection.

There are inconsistent conclusions about the impact of calibration data on Post-Training Quantization (PTQ) across studies, specifically regarding the factors that make the method effective, such as language that we explore in this work. We investigate this question by evaluating GPTQ and its mixed-precision variants across three low-bit settings, using language-targeted calibration sets across five language families. By evaluating performance across both perplexity and downstream tasks, we find that selecting an appropriate calibration language is generally beneficial, though model sensitivity varies with bit-width, target language, and task difficulty. We further show that mixed-precision techniques expand the representational capacity of low-bit models, occasionally allowing 2-bit models to approach 3-bit performance when paired with language-targeted calibration, while increased effective-bits remains small. Furthermore, we explore the correlation between these factors and observe that the calibration set sensitivity is influenced by different combinations of factors for each controlled variable, which can be explained by the downstream task results. Finally, while the benefit of language-targeted calibration transfers to sibling languages within the same family, the magnitude of this transfer does not reliably predicted by their closeness. Together, these findings show calibration data selection as a deliberate design choice in multilingual PTQ rather than an incidental one. Our code and result data can be accessed in https://github.com/rafifpriyo/multilingual-calibration-study.

Interpretable linguistic features offer a promising approach for explaining *why* a given text appears machine-generated, particularly for non-expert users. However, existing findings on which features reliably indicate LLM-generated text remain fragmented across feature sets, models, and text domains. To address this gap, we conduct a large-scale empirical study assessing the robustness of linguistic signals for characterizing AI-generated text. Our analysis covers 284 interpretable linguistic features across outputs from 30 LLMs and 10 text domains under cross-model and cross-domain generalization settings. We show that classifiers based solely on linguistic features can reliably distinguish AI-generated from human-written text. However, many previously proposed indicators prove strongly context-dependent, with the exception of measures of lexical richness, which remain robust signals across model families and text domains. These results demonstrate which linguistic signals generalize across contexts and provide a foundation for more reliable, interpretable analyses of AI-generated language.

Rubric-based evaluation is a promising approach for assessing open-ended outputs from LLM-based research agents, particularly in paper reproduction, where direct paper-to-repository comparison is prone to hallucinations. However, constructing paper-specific rubrics requires substantial expert effort, limiting the scalability of benchmarks such as PaperBench. In this work, we present, to our knowledge, the first systematic meta-evaluation of LLM-generated rubrics for paper reproduction. We reformulate rubrics into a checklist-style format and evaluate four generation settings across two backbone models. We meta-evaluate generated rubrics intrinsically by semantic similarity and extrinsically by score alignment with ground-truth rubrics. Our results show that the augmented settings substantially improves downstream evaluation alignment, with the strongest setting approaching the human baseline, while intrinsic gains are more modest. Further analyses reveal that LLM-generated rubrics are often overly fine-grained, biased toward high scores, and less adaptive to paper domains, highlighting both the promise and limitations.

Language models frequently generate outputs in unintended languages or scripts—a phenomenon known as off-target generation. While existing research has focused on language selection, the dimension of script knowledge remains understudied: before any linguistic understanding can occur, users must recognize the graphic symbols in a model’s response. We investigate whether Small and Large Language Models (SLMs and LLMs) possess script knowledge by testing them on multi-scriptic languages. Through two complementary experiments, we evaluate whether models (1) adapt their output script to match the input, and (2) follow explicit instructions to generate text in a specified script. The models we tested demonstrate substantial script knowledge: they all achieve a near-perfect Latin script fidelity (more than 98%), and follow script instructions with high frequency. Nevertheless, we notice differences between LLMs and SLMs, with higher scores for LLMs including for non-standard script combinations.

Graphical User Interface (GUI) agents powered by Multimodal Large Language Models (MLLMs) promise human-like interaction with software applications, yet long-horizon tasks remain challenging due to memory limitations. Existing approaches either truncate history or rely on simple textual summaries, which risk losing critical information when past visual details become necessary for future decisions. In this paper, we propose PAL-UI (Planning with Active Look-back), a novel framework that enables GUI agents to adaptively retrieve past observations when required. PAL-UI combines a dual-level summary agent, capturing both observation-level cues and action-level outcomes, with a dedicated retrieval tool that allows the agent to recall specific historical screenshots during planning. We curate a step-level instruction dataset of 8.6K samples from mobile GUI navigation trajectories and train PAL-UI-3B and PAL-UI-7B models based on Qwen2.5-VL. Extensive experiments show that PAL-UI significantly outperforms baseline models and prior methods in mobile GUI tasks, even under data-efficient settings. Moreover, PAL-UI exhibits strong cross-domain generalization, achieving notable improvements in web navigation without additional training. Our work highlights the potential of active memory retrieval for long-horizon planning capabilities of vision-based GUI agents.

Sinhala Sandhi splitting recovers the constituent words or morphemes hidden inside a phonologically merged surface form. The task is important for Sinhala NLP because Sandhi obscures lexical boundaries, but no prior published work has established a neural benchmark for Sinhala Sandhi splitting. We present a character-level sequence-to-sequence study based on SandhiLex, using native Sinhala Unicode input and evaluating recurrent encoder-decoder models for affixational and more complex lexicalized, derivational, and etymological Sandhi. A bidirectional LSTM encoder with a unidirectional LSTM decoder yields the best results on the hard subset, achieving 68.40% exact-match accuracy and 82.08% character-level accuracy. On the full affixational subset, the same model achieves 94.00% exact-match accuracy. Ablations show that bidirectional encoding is the largest contributor to performance, while native Sinhala script improves exact match accuracy over romanized input. Qualitative analysis indicates that many errors are near misses involving boundary adjacent characters or plausible but incorrect phonological substitutions. These results establish an empirical baseline for Sinhala Sandhi splitting and identify data scale, Sandhi type conditioning, and attention-based decoding as the main directions for future work.

Contrastive vision-language models have achieved remarkable progress through large-scale pretraining. Recent work has shown that removing English-only caption filters and pretraining on global data is effective for improving multicultural performance. We study whether such global pretraining is sufficient for culture-specific understanding, or whether further adaptation with natively sourced data can boost performance beyond what global pretraining alone achieves. To enable this investigation, we present WAON, the largest publicly available native Japanese image-text dataset constructed from native Japanese web content in Common Crawl, containing approximately 155 million examples. We also introduce WAON-Bench, a manually curated Japanese cultural benchmark spanning 374 classes. Through comparative fine-tuning experiments on multiple Japanese image-text datasets, we observe that models fine-tuned on WAON consistently achieve stronger performance on Japanese cultural benchmarks than those fine-tuned on English-to-Japanese translated data. Controlled experiments at matched scale, filtering, and training budget across two model families further indicate that native web origin is the primary driver of this gain. We release our dataset, benchmark, model, and code.

Retrieval-augmented generation is widely used in professional writing, yet whether retrieval grounds revision or merely injects templates is rarely tested where "correct" has a definable meaning. Patent claim amendment supplies that signal: the examiner names the attacked limitation and cites prior art, providing per-case ground truth. We release three artifacts: (i) a corpus of 7,385 USPTO prosecution cases with XML-aligned pre/post claims, rejection, and cited prior art; (ii) a seven-probe battery comparing random and structural-match retrieval as two policies under a fixed prompt scaffold; (iii) a deterministic five-channel metric (C1-C3 and C5 in main, C4 supplementary) requiring no LLM evaluation. Across 9,600 pre-registered calls on four frontier LLMs (Claude Sonnet 4, Claude Haiku 4.5, GPT-5.4, GPT-4o-mini), no tested model exhibits detectable classical prior-injection behavior; retrieval effects are small and direction-inconsistent between random and structural retrieval, and the null is unchanged under a dense (semantic) retriever, across retrieval depths k in 1,3,5,10, and under a paraphrase-sensitive grounding metric. Revision locality reveals a model-specific difference that the template channel misses. The four-cell taxonomy, which we treat as exploratory, leaves the prior-injector cell unoccupied.

Prompting has transformed large language models into general-purpose decision systems, where token-level probabilities are often treated as indicators of model preference and correctness. However, directly interpreting these probabilities is challenging due to calibration disparities, tokenizer variability, and inconsistent output distributions across models, limiting their statistical comparability. This raises a fundamental question: *when can a model be said to be confident in its decision or preference?* Addressing this question remains an open and important area of research. We introduce **PLAUSIEVE**, a novel distribution-based statistical framework that estimates model preference through a forward pass without token generation. **PLAUSIEVE** provides a calibration-robust and model-independent mechanism for quantifying preference without relying on fixed thresholds or individual tokens. We apply **PLAUSIEVE** in estimating model belief, across two prompt settings (direct and epistemic), two confidence levels, and five models from four families to study prompt–architecture interactions. Our analysis shows significant improvements under epistemic prompting, with nearly 100% gains in certain model combinations. Notably, **Llama-3.2-3B** outperforms the larger **Gemma-7B** under epistemic prompting in terms of both decisiveness and veracity, demonstrating that model architecture plays a more significant role than scale in shaping belief formation.

This work investigates whether Chain-of-Thought (CoT) prompting benefits patent claim generation. We demonstrate a counter-intuitive but important empirical finding: implicit CoT, where reasoning is kept internal rather than explicitly verbalized, consistently outperforms explicit CoT. Through systematic analysis, we show that explicit CoT can introduce an unnecessary information bottleneck for claim generation. Verbalized reasoning may compromise the quality of final outputs through three specific mechanisms: abstraction of critical details, disruption of internalized generation patterns, and cascading error propagation. Our findings provide new insights into structured text generation and CoT applications.

When LLMs annotate subjective content such as predicates of personal taste (PPTs), e.g., "Eliza tasted the cookie. It tasted delicious", a consequential but underexplored question is whose perspective they adopt: the narrator’s (default) or Eliza’s (shift toward the character)? Humans tend to attribute "delicious" to Eliza because tasting implies first-hand experience. We introduce GRANI (Graded Response Analysis for Narrative Implications), a protocol for evaluating LLMs’ graded understanding of such phenomenon. Applying GRANI to PPTs, we test whether text-trained LLMs reproduce human perspective shifts across sensory modalities in English and Chinese. We found that instruction-tuned models partially capture the human pattern, in which taste yields the strongest shift toward the character, vision the weakest, and smell an intermediate effect, while reasoning-distilled models are less consistent. Moreover, some models encode modality distinctions internally even when their behavioral responses do not. These findings suggest that subjectivity inference can emerge from text exposure alone, while highlighting a reliability concern: LLMs may not consistently adopt the intended perspective-holder when evaluating subjective content.

Synthetic supervision can bootstrap semantic parsers in new search domains, but prompt-induced diversity is useful only when generations are faithful to the structured intent. We explore this for cold-start e-commerce search: 118,098 synthetic pairs produced by 6 LLMs with 9 persona-conditioned prompt families on a 47-field UK automotive schema, against 1,214 human-annotated and 15,304 production queries. We show that the IPIP personality bundle improves parser F1 in all four matched contrasts (Holm-corrected p < .001), and all three bundles have a positive main effect over the complete 2^3 grid. Syntax conditioning is the axis that moves generations towards real traffic. It also induces lexical diversity. Alignment sKL, the divergence between von Mises-Fisher distributions fitted to synthetic and human-annotated queries, is an unlabelled proxy for parser F1 (rho = -0.917 on intent-elicited human queries, lower is closer). An LLM committee, checked against a blinded author pass, triages the 927 queries missed by every parser and labels 37% as hallucinated. Dropped constraints dominate. The four syntax-bearing families have the largest hallucinated shares, 38 to 42%, while IPIP alone has the lowest at 28%.

Chain-of-thought (CoT) prompting improves performance on complex reasoning tasks, but often triggers unnecessary computation on simpler queries. A natural approach is to use the model’s own confidence as a training-free signal to gate this decision, answering directly when confident and invoking CoT only when uncertain. Across four training-free confidence signals, three model families, and seven reasoning benchmarks, we find that the effectiveness of confidence-gated CoT varies, and it is unclear how to predict whether it will work for a new model. We address this with two key contributions. First, we identify that gating fails in the following circumstances: signal saturation, where confidence scores collapse to a narrow range and lose discriminative power; and selection bias, when confidence-gated routing sends the longest queries to CoT. Second, we propose a novel Pareto-optimal threshold selection method. This has the dual ability to preserve accuracy while reducing costs where the confidence signal is strong, and maintains accuracy where it is weak. Where the confidence signal is strong, gating reduces CoT usage by 30% without sacrificing accuracy. These findings provide practical guidance for evaluating and deploying models that selectively use CoT.

Multimodal sentence embedding models typically leverage image-caption pairs in addition to textual data during training. However, such pairs often contain noise, including redundant or irrelevant information on either the image or caption side. To mitigate this issue, we propose MCSEO, a method that enhances multimodal sentence embeddings by incorporating fine-grained object-phrase alignment alongside traditional image-caption alignment. Specifically, MCSEO utilizes existing detection and segmentation models to extract accurate object-phrase pairs, which are then used to optimize a proposed contrastive learning objective tailored to object-phrase correspondence. Experimental results on semantic textual similarity (STS) tasks across different backbone models demonstrate that MCSEO outperforms image-caption-only baselines on average, and that it is complementary to existing methods: combining MCSEO with KDMCSE yields further gains. A random-pairing control and a manual evaluation of the extracted pairs additionally indicate that the improvements stem from correct object-phrase alignment.

Multimodal fake news detection requires models to reason over both textual and visual evidence while remaining robust to distribution shifts in social media graphs. Existing approaches often concatenate modality-specific representations or rely on post-hoc fusion, which can leave image and text features in misaligned semantic spaces. Graph-based methods further introduce structural bias: node popularity, degree, or connectivity may become shortcut signals that correlate with labels in the training distribution but do not causally determine veracity. We present CIL, a causal-intervention and LLM-synergized framework for multimodal fake news detection. CIL first uses CLIP to obtain aligned image and text representations and constructs a multi-view semantic graph from cross-modal, intra-modal, and fused similarities. It then uses a large language model to produce high-confidence pseudo labels, which are injected as weak supervisory anchors. Finally, CIL mitigates degree-induced structural confounding with inverse propensity weighting and enforces counterfactual consistency under edge-drop interventions. Experiments on Twitter, PHEME, and Weibo suggest that each component contributes to performance, with the largest gains coming from aligned multimodal representations and causal debiasing.

Automated Essay Scoring (AES) for English proficiency assessment increasingly relies on pretrained transformer models, yet these models are typically trained on general-domain English and may under-represent second-language learner writing. This study investigates whether domain-adaptive continued pretraining (DAPT) on a learner-writing corpus improves transformer-based AES for English proficiency assessment. We perform DAPT on BERT, RoBERTa, and DistilBERT using the EFCAMDAT corpus, then compare the adapted models with their original checkpoints on two English proficiency test datasets, FCE and IELTS, in both in-domain scoring and few-shot cross-dataset transfer. Full-corpus DAPT produces mixed effects across models, datasets, and metrics. Subsequent lexical and syntactic analyses suggest mismatches between EFCAMDAT and the downstream datasets in proficiency level, genre, and communicative purpose. We therefore repeat DAPT using proficiency-specific EFCAMDAT subsets across all three encoder architectures. Proficiency-specific DAPT frequently outperforms full-corpus DAPT and, in some settings, even the non-adapted baseline. Overall, continued pretraining on learner writing can improve in-domain AES, but its benefits depend on both the proficiency composition of the pretraining data and the underlying encoder architecture, and do not consistently extend to cross-test transfer.

Evaluating conversational therapist competence in Motivational Interviewing (MI) requires multi-session assessment, yet current benchmarks rely on simulated patients that fail to model progression from session to session. We present MuSe-MI (Multiple Sessions Motivational Interviewing), a simulated patient framework grounded in Motivational Enhancement Therapy (MET), a multi-session clinical protocol of MI. MuSe-MI regulates dialogue generation through four evolving internal states from the MET clinical manual (anger, readiness to change, problem recognition, and self-efficacy), paired with an inter-session life event simulator and structured memory. Blind evaluations by licensed clinical psychologists confirm that patient dialogue reflects these internal states with high inter-rater agreement. Experiments show that MuSe-MI reliably distinguishes conversational therapist MI adherence levels based on patient language progression, establishing an automated multi-session evaluation benchmark.

While large language models (LLMs) have achieved remarkable linguistic capabilities, a profound question lingers at their core: do these models truly comprehend context or simply excel at pattern matching on an unprecedented scale? Contextual understanding in LLMs refers to the ability to correctly extract relevant information from a given context, integrate it into a coherent internal representation, and reason over it to produce factually consistent and contextually grounded responses. However, traditional methods such as BiLingual Evaluation Understudy (BLEU) and perplexity simply measure surface-level performance. This reveals a critical gap in question answering (QA), where responses must be contextually grounded rather than simply being memorized associations. To fill this void, we propose a novel knowledge graph (KG) based evaluation framework for LLM contextual understanding in QA. Central to this is Semantic Structural Similarity for KGs (S3KG), a hybrid similarity measure combining structural and semantic signals into a single score. In addition, a diagnostic analysis framework is developed to identify and categorize reasoning errors at the triplet level, enabling fine-grained analysis of model failures. Together, across nine benchmarks, S3KG achieves F1 gains of up to +7.6 points over the strongest baseline and AUROC up to 0.973.

Large Language Models (LLMs) are powerful tools for answering user queries, yet they remain highly vulnerable to jailbreak attacks. Existing guardrail methods typically rely on internal features or textual responses to detect malicious queries, which either introduce substantial latency or suffer from randomness in text generation. To overcome these limitations, we propose SelfGrader, a lightweight guardrail method that formulates jailbreak detection as a numerical grading problem using anchored token-level logits. Specifically, SelfGrader evaluates the safety of a user query within a compact set of numerical tokens (NTs) (e.g., 0–9) and interprets their logit distribution as an internal safety signal. To align these signals with the target safety rubric, SelfGrader constructs Probably Approximately Correct-guided ICL anchor examples and introduces a dual-perspective scoring rule that considers both the maliciousness and benignness of the query, yielding a stable and interpretable score that reflects harmfulness and reduces the false positive rate simultaneously. Extensive experiments across diverse jailbreak benchmarks, adaptive attacks, benign prompt benchmarks, multiple LLMs, and state-of-the-art guardrail baselines demonstrate that SelfGrader achieves strong robustness with low false positive rates, memory overhead, and latency. Warning: Some content generated by LLMs may be offensive.

Selective retrieval aims to make retrieval-augmented generation (RAG) more efficient and reliable by skipping retrieval when an LLM’s parametric knowledge suffices. Despite promising results, existing methods are constrained by a binary design choice: either retrieve from a single external source or skip retrieval and let the LLM directly produce the final answer. We argue that this fallback underestimates the model’s knowledge and obscures the more general multi-source decision problem that arises in practical systems. We propose Self-Routing RAG (SR-RAG), which casts selective retrieval as knowledge source selection and treats the LLM itself as a first-class knowledge source. SR-RAG learns to select an appropriate knowledge source, optionally verbalize parametric knowledge, and answer using the selected source, all within a single left-to-right generation pass. SR-RAG further augments source selection by combining LLM-based uncertainty with a flexible external policy datastore to improve decision calibration. Across four benchmarks and three 7B-class LLMs, SR-RAG outperforms a strong selective retrieval baseline by 8.5%/2.1%/4.7% while performing 26%/40%/21% fewer retrievals, and it achieves favorable accuracy-latency trade-offs without dataset-specific threshold tuning.

Manchu is a typical low-resource agglutinative language, for which existing digital resources are largely limited to character recognition and basic annotation, while sentence-level generation remains underexplored. In this paper, we study constrained sentence-level Manchu continuation. We construct the first context-target corpus for this continuation task, consisting of 3214 examples collected from Manchu Stories in Translation, Common Manchu Dialogues, and other sources, covering both narrative and conversational genres. To improve robustness to rich suffixation, relatively flexible word order, and frequent OCR or transcription noise in Manchu, we propose a Multi-Granularity Data Augmentation (MGDA) strategy that expands training data through character-level noise injection, phrase rearrangement, and suffix perturbation, without relying on external dictionaries or translation models. Experiments on ByT5-small and Qwen2-1.5B show that MGDA substantially improves Manchu text generation performance: BLEU-4/ROUGE-L increases from 2.03/2.59 to 17.60/42.70 for ByT5, and from 1.98/6.42 to 14.12/41.49 for Qwen. Further ablation studies demonstrate the complementarity of the three augmentation granularities and suggest potential benefits of byte-level modeling in this experimental setting.

Knowledge Graph Question Answering (KGQA) is particularly challenging for Persian, a complex and low-resource language with significant ambiguity. To address this, we introduce PeMA-KGQA (Persian Multi-Agent Knowledge Graph Question Answering), the first multi-agent LLM framework for Persian KGQA, arguing that an agent-based approach is essential for robust reasoning. Our system operates on FarsBase and is evaluated on both the ParsSimpleQA (one-hop) and PeCoQ (compositional) benchmarks. The framework employs a plan-and-execute paradigm where a dedicated Planner agent first decomposes a question into a structured executable plan using a toolbox of operators. Subsequent specialized agents—an Entity Grounder, a Relation Selector, and a Path Scorer—then execute this plan, effectively grounding mentions, selecting relations, and scoring reasoning paths to produce the final answer. This multi-agent decomposition provides a transparent and interpretable reasoning trace. PeMA-KGQA sets a new state-of-the-art on ParsSimpleQA with 21.14% improvement in Hits@5 and achieves 80.67% in Hits@5 on the complex multi-hop questions of PeCoQ.

Large language models pre-trained on code (Code-LLMs) have achieved remarkable performance on coding tasks. Despite their success of processing programming language, their application to natural language tasks has largely been limited to narrow tasks and ad-hoc model selection. In this paper, we present Code-Prompt, a comprehensive prompt-based framework to benchmark Code-LLMs over ten NLP tasks. We evaluate Code-Prompt on 13 Code-LLMs using code-aligned prompts and compare them with their parallel natural language LLMs (NL-LLMs) using natural language prompts.

Our results show that Code-LLMs perform on par with NL-LLMs across a range of NLP tasks, while producing more format-consistent generation with less redundancy. Our cross-lingual experiments further indicate that knowledge in Code-LLMs is well shared among different programming languages, whereas performance across different human languages still varies significantly. All our code and data will be made publicly available at https://github.com/chen-yuxuan/code-prompt.

Time and event expression extraction are fundamental temporal reasoning tasks, but the problem remains difficult due to annotation ambiguity, domain sensitivity, and unstable model behavior. Existing evaluations focus on in-domain performance, offering limited insight into reliability under distribution shifts. We evaluate multiple model configurations across families, architectures, and reasoning strategies over four dimensions of generalization, examining transfer from base performance, cross-dimensional correlations, and the effects of scale, architecture, and prompting. This provides a systematic study of how prompted LLMs generalize in time and event expression extraction tasks. We find that strong base-task performance generally predicts better generalization. However, this relationship weakens under substantial distribution shifts. Inductive prompting performs most consistently across domain shift, adversarial perturbations, compositionality, and length increase, while gains from scale, architecture, and deductive and abductive prompting strategies are uneven and dimension-specific. We conclude that LLM generalization in temporal extraction tasks cannot be predicted from any single dimension alone and cannot be reliably inferred from in-domain or single-dimension evaluations, highlighting the need for reasoning strategies that generalize across dimensions.

Similarly to humans, LLMs are susceptible to the anchoring effect in price negotiations, which can lead to unfavorable or unintended outcomes. To ensure LLM safety in dialogue tasks such as negotiation, a mechanistic understanding and mitigation methods are essential. We propose four variants of a method to identify Anchoring Heads, which are the attention heads responsible for the anchoring effect, while preserving those essential for general tasks. Experimental results show that suppressing these heads mitigates the anchoring effect by over 16%, achieving complete mitigation in certain models without compromising general performance. Furthermore, the observation that suppressing random heads fails to mitigate the anchoring effect suggests that specific attention heads drive the anchoring effect by paying excessive attention to the anchor.

Topic models summarize large text corpora, but top-ranked words often provide only a limited representation of topic semantics. Sparse autoencoders (SAEs) offer a way to move beyond word-level descriptors by extracting interpretable features from dense representations, yet how feature interpretability relates to topic-inference quality remains unclear. We introduce MonoTM, an interpretable topic modeling framework that decouples these roles. Across three benchmark corpora, we show that document–topic mixture estimation and semantic interpretation favor different SAE configurations and feature subsets. MonoTM estimates mixtures from the full SAE bag-of-features representation and, with them fixed, learns topic descriptors over a separate vocabulary of corpus-grounded semantic features. This design preserves global topic structure while representing topics with semantic units more meaningful than individual words, making them more useful for downstream corpus analysis.

Tool use, such as web search, has become a standard capability even in freely available large language models (LLMs). However, existing benchmarks evaluate temporal reasoning mainly in static, non-tool-using settings, which poorly reflect how LLMs perform temporal reasoning in practice. We introduce Time Puzzles, a constraint-based date inference task for evaluating iterative temporal reasoning with tools. Each puzzle combines factual temporal anchors with (cross-cultural) calendar relations and may admit one or multiple valid dates. The puzzles are algorithmically generated, enabling controlled and continual evaluation. Across 13 LLMs, even the best model (GPT-5) achieves only 55.3% accuracy without tools, despite using easily searchable facts. While web search improves performance, models perform substantially better when constraints are rewritten with explicit dates, removing the need for factual lookup. These results reveal a gap in reliable tool use for iterative temporal reasoning.

While the field of vision-language (VL) has achieved remarkable success in integrating visual and textual information across multiple languages and domains, there is no dedicated framework for assessing human-centric alignment in vision-language systems.

We offer two contributions to address this gap. First, we introduce Anthropogenic Regional Adaptation: a regional-adaptation framework that aims to optimize model relevance to specific regional contexts while ensuring the retention of global generalization capabilities. Second, we present a simple, but effective adaptation method named Geographical-generalization-made-easy (GG-EZ), which utilizes regional data filtering and model merging.

Through comprehensive ablations on 3 VL architectures: large vision-language models (LVLM), text-to-image diffusion models, and vision-language embedding models, and a case study in Southeast Asia (SEA) regional adaptation, we demonstrate the importance of Anthropogenic Regional Adaptation and GG-EZ, showing 5-15% gains in cultural relevance metrics across SEA while maintaining over 98% of global performance and even occasionally surpassing it.

Our findings establish Anthropogenic Regional Alignment as a foundational paradigm towards applicability of multimodal VL architectures in diverse regions and demonstrate a simple-yet-effective baseline method that optimizes regional value alignment while preserving global generalization.

Large language models (LLMs) can achieve strong performance on discourse relation prediction through fine-tuning. Yet it remains unclear whether such performance reflects a robust modelling of inter-sentential relations or a reliance on shallow lexical cues. This paper presents a behavioural audit of fine-tuned models across explicit and implicit settings. In explicit discourse relation prediction, we find that fine-tuned models rely heavily on connectives as categorical triggers, rather than integrating argument semantics. More surprisingly, in implicit discourse settings, we observe a severe performance degradation when masking semantically light function words from the arguments, suggesting that models may also exploit subtle surface-level cues beyond explicit connectives. These results suggest that high accuracy alone is insufficient to demonstrate robust discourse processing. To address this limitation, we propose a causally validated diagnostic framework that measures whether model predictions are grounded in linguistically meaningful evidence. Our metric provides a complementary perspective to standard accuracy-based evaluation and establishes a rigorous auditing toolkit for future work on discourse relation prediction.

LLM safety guardrails often behave unevenly in localized multilingual settings, especially with code-mixed vernaculars and non-Latin scripts. We introduce RabakBench, a Singapore-localized multilingual safety benchmark for Singlish, Chinese, Malay, and Tamil, built through a semi-automated, human-validated Generate–Label–Translate pipeline. The benchmark contains 1,341 unique examples in four language versions, yielding 5,364 entries across six harm categories, with LLM-assisted labels validated against human consensus. Evaluating 13 commercial and open-source guardrails reveals substantial cross-language and system-level variation, highlighting localized and script-sensitive safety gaps in the Singaporean context. RabakBench provides both a diagnostic resource and a reproducible, human-in-the-loop framework for constructing localized safety benchmarks.

Large language models (LLMs) achieve strong performance across many tasks, but their high computational cost limits deployment in resource-constrained environments. Knowledge Distillation (KD) offers a practical solution by transferring knowledge from a teacher model of a larger size to a smaller student model. While prior work has mainly examined task-specific or small-scale settings, the post-training stage for building general instruction-following models has received limited attention. In this paper, we conduct a systematic study of KD in post-training using the large-scale Tulu 3 dataset. We find that KD outperforms supervised fine-tuning (SFT) in low-data regimes, but its advantage diminishes as more training data is added. Distilling from a stronger instruction-tuned teacher restores substantial gains even with abundant data, indicating that KD remains effective when the teacher provides knowledge that the student cannot easily acquire from the training data alone. We further study domain-specific, low-resource scenarios and propose a two-stage KD strategy that leverages synthetic teacher-labeled data followed by refinement on human annotations. This method consistently improves student performance, providing practical guidance for building compact models in data-scarce environments.

This study proposes a new LLM-as-a-judge method for relative evaluation of LLM responses based on both required criteria (i.e., minimal conditions that responses must satisfy) and bonus criteria (i.e., desirable additional qualities). Recent LLM-as-a-judge methods have improved the stability and accuracy of relative evaluation through the use of automatically generated criteria. However, existing methods rely solely on the required criteria, which may limit their ability to distinguish closely matched responses. To address this issue, the proposed method first evaluates responses based on required criteria and, in the case of a tie, further evaluates them using bonus criteria, enabling more fine-grained comparisons. Experiments on the Writing, Roleplay, and Humanities categories of the MT-Bench dataset demonstrate that the proposed method achieves higher agreement with human judgments than existing methods.

Singlish is a creole rooted in Singapore’s multilingual environment that continues to evolve alongside social and technological change. We examine diachronic stylistic change across a decade of informal digital messages and ask whether Large Language Models (LLMs) can generate temporally neutral outputs approximating the stable characteristics of the variety. Using lexical, pragmatic, psycholinguistic, and encoder-based features, we find that stylistic separability increases with temporal distance, driven primarily by structural features such as length and complexity. Evaluated against a null distribution baseline, most LLMs fail to achieve both authenticity and temporal neutrality simultaneously at their most authentic prompting configuration. This reveals a structural trade-off: models generating realistic Singlish tend to inherit its temporal biases, while temporally neutral models tend to produce inauthentic outputs. These findings position temporal neutrality as a diagnostic metric for assessing sociolectal grounding in LLMs.

Political debates shape public opinion, but many readers increasingly encounter debate takeaways through AI-generated summaries and commentary rather than full broadcasts. This paper studies whether fine-grained lexical choices in debate transcripts can influence the evaluative framing of GPT-generated political commentary. We introduce controlled word-level interventions by replacing embedding-identified key lexical items with near-synonymous alternatives, allowing us to examine how minimally invasive lexical changes affect model-generated interpretation. Across U.S. presidential debate segments, we find that single-word substitutions can shift the stance of GPT-generated commentary, but these effects are probabilistic, context-dependent, and sensitive to the speaker’s communicative goal. Substitutions are more likely to produce favorable stance shifts in critical contexts than in promotional ones, suggesting that GPT-generated commentary may amplify adversarial framing more readily than self-promotional framing. We further analyze which linguistic cue types are associated with substitution success, including sentiment direction, part of speech, and contextual suitability, and show that sentiment alone is insufficient to explain model responses. Finally, a small exploratory human evaluation suggests partial alignment between model-based stance shifts and reader preferences in promotional cases, while also revealing substantial subjectivity in human judgments. These findings highlight how candidates’ word choices may be transformed when political discourse is filtered through GPT-generated commentary, and caution against treating model-internal stance metrics as direct substitutes for large-scale human audience evaluation.

Modern visual CAPTCHA has evolved from simple perception tests into real-world spatial reasoning challenges. It remains unclear whether state-of-the-art vision-language models can solve such tasks through their own reasoning ability. Existing benchmarks offer limited insight into this question as they often ignore intermediate reasoning steps, typically assess only final answers, and thus cannot know how models fail. To address this gap, we present CAPTCHA-X, the first unified evaluation framework for real-world CAPTCHA solving with reasoning steps, enabling automatic assessment of solving accuracy and reasoning quality. Using this framework, we show that reasoning substantially improves performance. Beyond evaluation, CAPTCHA-X exposes characteristic reasoning failure modes of vision-language models and reveals reasoning scaling laws that connect multiple dimensions of reasoning. Furthermore, evaluating pure vision generation models reveals an interesting phenomenon: generation models also fail at multi-step logical planning on images. These findings motivate our super-simple and reasoning-centric agent built solely on vision-language models, designed to identify and correct errors in intermediate reasoning steps before translating reasoning into executable actions.

Vision language models are predominantly evaluated on English-centric multimodal benchmarks, leaving their behavior in other languages and cultures insufficiently understood. We introduce VMMU, a Vietnamese Multitask Multimodal Understanding and Reasoning benchmark of 2,548 multiple-choice questions across seven domains: Mathematics, Physics, Chemistry, Biology, Geography, Driving Test, and IQ Test. Every question requires jointly reasoning over Vietnamese text and non-text visual evidence (charts, diagrams, tables, traffic scenes), and is distributed as a single rendered page image. We evaluate 10 open-source and 5 closed-source VLMs on VMMU. Test-time compute improves performance from 57% to 78% accuracy, but it remains inferior to an expert baseline (99% accuracy). Extensive error analyses show that closed-source VLMs already achieve strong Vietnamese OCR performance, yet still struggle on VMMU. This suggests that the primary bottleneck is multimodal grounding and reasoning rather than text recognition or Vietnamese language understanding. Code and data are available at: https://vmmu-bench.github.io.

Framing in news media shapes how events, actors, and causal responsibility are interpreted, yet computational framing analysis remains limited for lower-resource languages and fast-evolving international conflicts. In this work, we analyze narrative framing in news coverage of the 2025 Thailand–Cambodia border conflict across English, Thai, and Khmer news articles. To support benchmarking for this task, we manually annotate 51 conflict-related articles using a stakeholder-oriented Hero–Villain–Victim (HVV) framework and source attribution annotations. We benchmark proprietary and open-weight large language models (LLMs), compare original-language and translated input, evaluate salience-focused prompt variants, and test supervised cross-dataset transfer. Original-language inference and explicit article-level salience instructions improve the best micro-averaged F1 from 0.66 to 0.73. Applying this improved pipeline across active-conflict and post-conflict periods suggests largely mirrored partisan framing patterns in Thai and Cambodian national media.

Cultural adaptation in dialogue systems refers to the ability to generate responses that conform to linguistic norms and social expectations across different cultures. However, existing dialogue generation approaches pay limited attention to the role of personality in shaping cultural expression and communication style. We therefore propose HiCAP, a novel framework that integrates personality traits into culturally adaptive dialogue generation. HiCAP employs a personality-conditioned control mechanism that hierarchically modulates response style by first aligning it with the cultural tone and then refining linguistic expression and content according to individual personality attributes. To embed these factors, the model combines supervised fine-tuning and instruction-based tuning, supported by dynamic instruction templates and multi-agent demonstrations to capture diverse combinations of culture and personality. Experiments on CulturalPersonas and CPED show that HiCAP achieves the strongest overall performance across various dialogue generation metrics, demonstrating effective cultural adaptability and personality modelling.

Privacy auditing of large language models (LLMs) increasingly relies on advanced jailbreak attacks to uncover unintended disclosures of personal information; the reliability of such audits, however, hinges on evaluators that can accurately determine whether a response constitutes privacy leakage. This evaluation remains challenging because disclosures may be indirect, contextual, partial, or obfuscated, and different evaluators can interpret the same response differently. We conduct a systematic study of guard models and LLM-as-Judges and find substantial disagreement among evaluators, particularly in ambiguous cases that require additional evidence or targeted analysis to resolve. Motivated by these findings, we introduce PrivacyEvalClaw, a training-free deliberative agent for evaluating privacy leakage in LLM jailbreak responses. Rather than relying on isolated single-pass judgments, PrivacyEvalClaw treats evaluators and analysis modules as tools, adaptively gathers evidence, investigates uncertain disclosures, and reconciles conflicting signals before producing a final judgment. Our primary experiments use pairwise LLM-as-Judge evaluation of complete trajectories, showing that PrivacyEvalClaw achieves favorable win rates against competing baselines across outcome, process, and overall quality, while also producing explicit reasoning traces that make the evaluation process more transparent and auditable.

Standard RLHF pipelines often reduce heterogeneous human judgments into a single scalar reward target. We argue that this reduction can mis-measure alignment in structurally plural societies, where disagreement may reflect culturally, historically, linguistically, regionally, or normatively grounded interpretations rather than annotation noise. We call this failure Preference-Validity Compression, the collapse of multiple plural-valid response options into a single optimization target. Using Malaysia as a diagnostic setting, we analyze RLHF-style feedback aggregation through preference events linking prompts, responses, and acceptability judgments across interpretive frames. Across 321 preference events from 20 participants and 107 trio-annotated prompts, 79% of prompts contain more than one majority-supported response that single-winner aggregation would discard, and apparent dominance gaps between top responses diminish when all majority-supported options are considered. Participants frequently select multiple acceptable responses, and discarded responses demonstrably reflect coherent local, practical, or cultural frames. These findings show that majority aggregation in this corpus measures \argmax acceptability rather than plural alignment. We treat this as a measurement-validity issue and argue that future alignment methods should satisfy Validity-Preserving Consistency, remaining stable across plural-valid interpretive frames rather than collapsing them into a single reward target.

Prompt compression is aimed at reducing input prompt lengths to enable cheaper, faster, and potentially higher-quality LLM predictions. However, existing prompt compression methods are often limited by modest compression gains, a risk of hallucination, and/or high compression latency. We propose EffComp, an **Eff**icient prompt **Comp**ression framework for RAG-based question answering (QA). EffComp employs BERT-style document reranker and sentence selector models to allow fast extractive prompt compression at the sentence level. Its extractive nature prevents hallucinations in the compressed prompts. Additionally, the training process using a hybrid reinforcement and supervised learning approach is designed to optimize the compression ratio while preserving LLM accuracy. Experiments on various QA datasets demonstrate that EffComp outperforms state-of-the-art prompt compression methods achieving the best trade-off between compression ratio and accuracy with minimal latency. This makes it practical for real-world applications.

As we continue to extend their range of applications, the study of the generalization capabilities of large language models (LLM) has become an important topic. Previous works have recently proposed to study these abilities in the context of mathematical reasoning. This was done by asking models to reproduce proofs that they have most likely seen during training, but when key symbols are replaced. Since this evaluation relies heavily on human feedback, we study to what degree this evaluation can be automatically performed. To that end, we simulate the human evaluation proposed by previous work by interactively presenting the same questions to the LLMs as prompts. We analyze the degree of alignment between human and model answers via an accuracy and correlation analysis, as well as via inter-annotator agreement metrics. We test a selection of models, including ones with reasoning capabilities, and show LLMs currently have important limitations in this regard, which cannot be addressed by ICL.

Recent advancements in Large Language Models (LLMs) have shown remarkable capabilities in multilingual contexts. However, previous studies suggest that multilingual LLMs primarily rely on the rich semantic knowledge of high-resource (i.e., English) representations to conduct tasks across languages. To mitigate the performance gap arising from resource imbalances, we propose Steering via Language-Specific Neurons, a training-free method that aligns non-English representations with the English manifold intervening on language-specific neuron activations via linear interpolation toward English activation prototypes. Experimental results show that the proposed method generally improves downstream task performance with minimal computational cost, yielding performance gains for low-resource scenarios where limited data often constrains knowledge coverage. Our findings suggest that the fundamental limitations of low-resource languages can be effectively alleviated by leveraging the robust representation space of high-resource languages, offering a promising direction for more balanced multilingual LLMs.

Large Language Models (LLMs) are widely used because they process structures, syntax and code well, but this same ability also makes them paradoxically vulnerable. We introduce BreakFun, a jailbreak method that frames a harmful request as code-execution simulation. The prompt gives the model a benign Python class definition, the "Trojan Schema", and asks what that code would print if it ran. To answer, the model must create an object from the class and invent a value for each field, and the adversarial field names steer those values toward the attacker’s goal. The harmful content therefore appears as a side-effect of the simulated instantiation, not as a direct answer. A three-part prompt carries the attack: an innocent frame, the Trojan Schema, and a Chain-of-Thought distraction. On JailbreakBench, BreakFun reaches an average attack success rate of 89% across 13 open-weight and commercial models (open-weight ~98%, API systems ~78%), and 100% on several. An ablation identifies the Trojan Schema as the strongest contributing component. As a counter-ablation probe we introduce Adversarial Prompt Deconstruction (APD), in which a second LLM writes out all readable text in the prompt before it judges safety. APD improves detection across three model families, and a probe-specific ablation shows that the transcription step gives most of the gain. Our findings show that LLM safety must account for content that a simulation task produces as a side-effect, not only for content that a user requests directly.

LLM agents now operate as a privileged application layer, with access to mail, files and MCP-connected tools, yet they have no built-in way to verify that the person typing into an already-signed-in session is still the legitimate user. This identity gap aligns with OWASP LLM06 (Excessive Agency), Agentic ASI03 (Identity and Privilege Abuse), and NIST guidance on AI risk management, but whether it is closable from user text alone is largely open. We build a profile-query authorship verification benchmark on WildChat-1M and PAN-20 regrouped per author, with author-disjoint 60/20/20 splits across three seeds. We train contrastive bi-encoders with triplet margin loss on Distil- BERT, BERT, MiniLM-L6 and ELECTRA, under 100% random imposters or a 50/30/20 mix of random, semi-hard and hard imposters mined by TF-IDF. Models trained on random imposters look strong (ROC-AUC above 0.95) but drop by around 10 points on WildChat against topic-similar imposters, suggesting they rely on topic and vocabulary rather than writing style only. Our contribution is to expose and quantify this failure mode and to provide a reproducible hardness-stratified benchmark; applying established hardness-aware training as a controlled mitigation closes most of the gap, lifting TAR@FAR=1% on WildChat Hard from around 6–7% to 20–22% for BERT and DistilBERT with median messages-to-detect of one, and the same direction holds under a secondary SBERT miner. We release the benchmark construction and evaluation code.

Recent advances in artificial intelligence have seen natural language processing (NLP) applications used increasingly for mental health support, such as question-answering systems, conversational agents, and chatbots. However, these applications often require high-quality dialogue datasets, which are scarce for certain under-resourced mental health conditions like gambling disorders. Privacy concerns and the sensitive nature of mental health issues make real-world data collection difficult and time-consuming. We propose a framework that uses publicly available social media posts as seeds to generate empathy-driven multi-turn dialogues between help-seekers and supporters in mental health conditions that have been under-researched in NLP. Through a case study on gambling disorder, we demonstrate the effectiveness of our approach in producing higher-quality dialogues than standard role-playing prompts. Our work showcases an effective dataset generation method for fine-tuning mental health language models alongside downstream applications like dialogue system development, particularly for low-resourced mental health disorders like gambling disorders.

Large reasoning models (LRMs) extend large language models by generating explicit chain-of-thought (CoT) reasoning, significantly improving mathematical and logical problem solving. However, this explicit reasoning process also introduces new safety risks, as unsafe behaviors often emerge within intermediate reasoning trajectories, even when final answers appear harmless. Existing safety alignment approaches primarily rely on supervised fine-tuning (SFT) over safety-oriented long CoT datasets. While intuitive, we find that SFT produces inconsistent safety improvements, degrades reasoning ability, and generalizes poorly across model families. These limitations suggest that purely supervised approaches are insufficient for robust safety alignment in LRMs. Motivated by these limitations, we systematically compare post-training paradigms for LRM safety alignment, including SFT, preference-style optimization, and reinforcement learning (RL). Unlike SFT, RL directly optimizes model policies with reward feedback, enabling more adaptive and stable alignment. Extensive experiments across multiple model families and benchmarks show that RL achieves stronger and more consistent safety gains while maintaining reasoning competence. Further analysis of reflection dynamics and token-level entropy reveals that RL suppresses unsafe exploratory reasoning while preserving reflective depth, leading to safer and more reliable reasoning processes.

Tibetan, spoken by some seven million people on and around the Qinghai-Xizang Plateau, remains underrepresented in natural language processing (NLP). We present, to our knowledge, the first peer-reviewed systematic bibliometric survey of Tibetan NLP: 140 publications from 2004 to 2026 in the ACL Anthology and DBLP, annotated for venue type, research area, first-affiliation country, and resource release. Output has grown rapidly, with 84% of the publications appearing since 2018, and has expanded bottom-up along the NLP pipeline, from corpora and word segmentation through script, speech, and applications to large language models (LLMs). Researchers based in China first-authored 87% of the publications, which span all 25 represented research areas, whereas work from other countries concentrates on Classical Tibetan. We argue that this full-pipeline paradigm, realized under the compounded constraints of a complex script, dialectal diversity, and geographic remoteness, offers a reference model for other low-resource languages of the Global South. We also identify gaps in dialect coverage, responsible NLP, native and comparable evaluation, cross-community visibility, and resource release (only 29% of the publications release data, code, or models), and recommend open science, shared standards, and community participation for the LLM era.

Large language models (LLMs) have demonstrated strong capabilities in financial analysis and reasoning, inspiring recent advances in agent-based trading frameworks. While these systems show promise, prior approaches either emphasize long-horizon forecasting or operate as stateless analyzers, limiting their applicability to the demands of trading in complicated settings. To address these gaps, we introduce META (Memory Enhanced Trading Agent), the first RAG-like episodic-memory-augmented multi-agent framework for financial decision making. META integrates a family of specialized indicator agents (e.g., Trend, MACD, Stochastic, RSI, SMA, AVWAP, Heikin-Ashi) with a Decision Agent that fuses their reports, and a Memory module that retrieves and updates past trading episodes encoded as market state embeddings with outcomes and reflections. By recalling relevant experiences and adaptively reweighting signals under similar market regimes, META achieves improved directional accuracy and robustness under short-horizon evaluation. Our results demonstrate that episodic memory provides a powerful mechanism for regime-aware, interpretable, and low-latency decision-making in trading and decision making. Code of this project is released at GitHub.

Procrastination is driven by emotional self-regulation failures and cognitive distortions that narratives often mask. We introduce ProcrastiType-ZH, to our knowledge the first Traditional Chinese multi-dimensional procrastination dataset, with 1,000 LLM-generated narratives and human consensus labels for four motives on a 1-5 severity scale (effective range 1-4). We propose ProcrastiCoT, a CBT-informed Chain-of-Thought framework, and inject its LLM-generated reasoning into BERT-size encoders either by concatenation or through a Dual-Encoder Cross-Attention architecture that gives the reasoning its own context window. Across backbones, reasoning injection gives the best single-run QWK on Avoidant, Decisional, and Irrationality. On our primary CKIP-BERT backbone, the Dual-Encoder scores above its pure-encoder counterpart on every dimension in each of four paired training runs, significantly (uncorrected) on Arousal, Decisional, and Irrationality but not Avoidant; on Decisional its mean (0.316) only matches a lexical baseline. A Qwen2.5-7B model fine-tuned on the same labels, with about 70 times more parameters, is strongest on Arousal (QWK 0.709) but below the Dual-Encoder on Irrationality and Avoidant in all four runs. Adding the teacher’s reasoning to its targets lowers QWK on every dimension and pulls its predictions toward the teacher’s ratings, consistent with a conflict between that reasoning and the human labels.

Real-world healthcare data is inherently complex: noisy, incomplete, heterogeneous, and temporally irregular, making clinically robust AI development difficult. Traditional models, trained on idealized datasets, fail to capture these operational realities, limiting real-world performance. We propose the Messy Clinic paradigm: a dataset and blueprint for building AI systems grounded in authentic clinical complexity. This framework enables seamless multimodal data integration across EHRs, imaging, genomics, and wearables, using privacy-preserving methods such as federated learning, synthetic data, and differential privacy. It provides (1) structured methodologies for longitudinal multimodal management, (2) AI techniques resilient to noise and missing data, and (3) governance mechanisms for consent, accountability, and bias mitigation. By embracing the “Messy Clinic,” healthcare AI can shift from artificial idealism to real-world robustness, accelerating research, enabling personalized care, and aligning AI with the true complexity of clinical practice.

Large-scale agents increasingly rely on retrieval to access external capabilities. We study this retrieval gate in structured tools and APIs, a measurable class of tool-backed executable skills that must be surfaced before an agent can plan, incorporate, or act. In this setting the retrieval layer can silently fail even when the capability corpus is fixed: on ToolRet, a retriever fine-tuned on one source-specific slice collapses on another source-specific slice of the same benchmark, with FT-1100 falling to 0.7% coverage on APIGen despite its higher lexical overlap with the gold tools. We call this failure mode source-style collapse. Query-side TF-IDF centroid distance identifies low-coverage transfer sources more reliably than semantic- or length-based proxies, providing a lightweight compatibility signal over a fixed tool corpus. We propose ToolScout, a source-aware routing method that uses this signal as a routing criterion: on the mixed 4,996-query stream, TF-IDF-based routing raises coverage from 22.3% to 86.1%, and across five collapsed sources, 20-shot source-specific adaptation, using 20 training queries per source, raises the coverage-weighted global top-1 proxy from 1.3% to 53.9%. The same failure and routing behaviors persist under executable skill-card rendering, showing that source-style collapse also occurs with this capability representation.

Feedback presentation shapes how LLMs use evaluator judgments during translation refinement. We introduce a machine translation protocol that keeps the evaluator score, main issue, and recommendation fixed while varying feedback form and supporting detail. Across seven models and six translation directions, low-load feedback outperforms high-load feedback in all 42 translator-direction settings under xCOMET. On the 29,400-record rewriting subset, mean xCOMET scores multiplied by 100 are 83.30 for low-load feedback and 80.67 for high-load feedback; the full-matrix original-translation reference mean is 83.46. MetricX-24 supports the low-load advantage on a stratified subset. A no-feedback second pass yields a small improvement over the original translations, and direct evaluator prompting reproduces the low-load advantage without a separate rewriting stage. These results show that compact feedback limits the quality loss from evaluator-guided refinement in the evaluated high-resource MT settings.

Multi-label text classification is important for organizing large document collections, but correlated labels and skewed label distributions make representation learning difficult. Existing multi-label contrastive methods usually define pairwise supervision from label overlap, resulting in a coarse signal limited to observed shared labels. We propose Greedy Matching Aggregation Soft Contrastive Learning (GMA-SCL), which constructs contrastive targets for MLTC by greedily matching label texts and incorporating label importance weighting. The resulting reference distribution is sharpened to emphasize highly similar candidate documents while preserving graded supervision from moderately similar ones. Experiments on AAPD, Reuters-21578, and UK-LEX show that GMA-SCL improves Macro-F1 over existing multi-label contrastive baselines while maintaining competitive Micro-F1 and Hamming Loss. Our code is available at https://github.com/koba-84/gma-scl.

Bail decisions constitute a large share of criminal proceedings in Indian courts, yet they often suffer from delays, inconsistency, and limited transparency. More than 75% of India’s prison population consists of undertrial prisoners, highlighting the urgency of improving the efficiency of bail adjudication. In this work, we present the Indian Bail Prediction System (IBPS), an AI-assisted framework that predicts bail outcomes and generates legally grounded explanations from factual case attributes and statutory context. We construct a large-scale dataset of 89,969 bail judgments from 5 High Courts of India, enriched with structured annotations such as statutes, custody duration, criminal history, health conditions, and judicial reasoning. Using this dataset, we design a two-stage architecture that combines a transformer-based classifier for outcome prediction with a fine-tuned large language model for rationale generation. Our results demonstrate the feasibility of combining structured legal data and large language models to develop interpretable AI systems that support data-driven analysis of bail decisions.

Driven by surging submission volumes, scientific peer review has catalyzed two parallel trends: individual over-reliance on LLMs and institutional AI-powered assessment systems. This study investigates the robustness of "LLM-as-a-Judge" systems to adversarial PDF manipulation via invisible text injections and layout-aware encoding attacks. We specifically target the distinct incentive of flipping "Reject” decisions to "Accept,” a vulnerability that fundamentally compromises scientific integrity. To measure this, we introduce the Weighted Adversarial Vulnerability Score (WAVS), a novel metric that quantifies susceptibility by weighting score inflation against the severity of decision shifts relative to ground truth. We adapt 15 domain-specific attack strategies, ranging from semantic persuasion to cognitive obfuscation, and evaluate them across 13 diverse language models (including GPT-5 and DeepSeek) using a curated dataset of 200 official and real-world accepted, rejected, and template papers (e.g., ICLR OpenReview). Our results demonstrate that obfuscation techniques like "Maximum Mark Magyk" successfully manipulate scores, achieving an increase in acceptance rates of up to 86.26% in open-source models, while exposing distinct "reasoning traps" in proprietary systems. We release our complete dataset and injection framework to facilitate further research on the topic (https://github.com/Manishprsd563/scientific-reviewer-jailbreak).

Multi-turn agent trajectories often contain redundant rounds (failed tool calls, parallel sub-queries, verification-only steps) that inflate both training and inference cost. We propose viewing each trajectory as a round-level dependency DAG that exposes which rounds are globally load-bearing for the final answer, and fine-tune agents on trajectories refined through this DAG. Operating on the DAG lets us apply deterministic and interpretable edits, pruning rounds whose outputs are never consumed and merging interchangeable sibling rounds, with optional LLM-based rephrasing. Models trained on these refined trajectories consistently outperform those trained on the original trajectories at lower inference cost. Specifically, across four multi-modal QA benchmarks, our refinements improve downstream accuracy by up to ,pp over vanilla SFT (and ,pp over an LLM-deletion baseline) while reducing per-sample inference messages by up to and inference tokens by up to , translating to substantial savings in compute and serving cost. Code is released.

While Large Language Models (LLMs) perform well on arithmetic word problems, they remain brittle to adversarial perturbations like irrelevant distractors or contradictory facts. To address this, we introduce ReThink, a training-free neuro-symbolic framework that formulates adversarial arithmetic reasoning as a Weighted MaxSAT/MaxSMT problem. ReThink uses a four-stage pipeline: it explicitly decomposes problems to strip noise, encodes the mathematical core as confidence-weighted soft constraints in the Z3 theorem prover, and cross-verifies the formal solution against a parallel Chain-of-Thought (CoT) path. This formulation can sacrifice lower-weight conflicting statements while satisfying higher-weight constraints. Evaluated on a balanced 2,400-example subset of GSM-Plus, ReThink with GPT-4.1 achieves 81.62% adversarial accuracy, outperforming CoT, PAL, and Self-Verification. It narrows the robustness gap to 14.69 percentage points while achieving 96.31% clean accuracy.

Unlike conventional static graphs, dynamic graphs evolve over time as their links change. Recent benchmarks, such as the Dynamic Text-attributed Graph Benchmark, further enable the study of dynamic graph learning with rich textual information on nodes and edges. The success of graph instruction tuning on static text-attributed graphs motivates us to extend this approach to dynamic graphs. However, previous works suggest directly using LLMs as predictors on dynamic graphs often underperforms traditional dynamic graph models. To address this issue, we propose a novel two-stage training framework consisting of Dynamic Text-attributed Graph Instruction Tuning (DTGIT) followed by standard supervised training. Unlike prior approaches that employ instruction tuning to enhance LLMs’ understanding of graph structures and directly use LLMs as predictors for downstream tasks, our proposed instruction tuning framework serves as a warm-up stage before standard supervised training. Specifically, during the instruction tuning stage, the embeddings produced by the dynamic graph model are aligned with temporal semantics in the natural language space, while the LLM itself is not directly used as a predictor for downstream tasks. In the second stage, the dynamic graph model is initialized with the knowledge acquired during instruction tuning and further fine-tuned in a supervised manner for downstream tasks. Extensive experiments demonstrate that existing dynamic graph models (e.g., DyGFormer) can achieve consistent performance improvement on the downstream tasks, particularly on link prediction, when trained with our framework.

Bias in large language models (LLMs) remains a persistent challenge, often leading to stereotyping and unfair treatment across social groups. While prior work has mainly focused on individual LLMs, the emergence of multi-agent systems (MAS), where multiple LLMs collaborate and communicate, introduces new and underexplored dynamics in how bias emerges, propagates, and amplifies. To systematically investigate these dynamics, we propose a simple evaluation framework with three agent-level metrics that quantify bias emergence, propagation, and amplification throughout multi-agent interaction. We evaluate MAS across three bias benchmarks under varying LLM backbones, social-group configurations, communication behaviors, and adversarial settings. Our results show that communication can trigger up to 70% new bias emergence, propagate bias across over 80% of agents, and amplify stereotypes by more than 3x. We further find that denser and competitive communication generally increases bias. Finally, we demonstrate that MAS are highly vulnerable to simple bias injection attacks, and existing defense strategies provide only limited protection. Our findings provide important insights into the fairness and robustness of multi-agent LLM systems.

Political polarization on social media intensifies during elections, when echo chambers harden partisan divisions and amplify online harms at the moment collective decisions are being made. Classical opinion dynamics models such as Friedkin-Johnsen provide a framework for studying belief change but abstract away the textual content that drives opinion shifts on social platforms. LLM-powered agent-based models reintroduce language to simulation, yet their fidelity to real political discourse—as opposed to synthetic personas or factual-consensus domains—remains untested against classical baselines. We construct a benchmark from the 2024 Indonesian Presidential Election: ~1,000 agents, 70,336 interactions, and five topics spanning four weeks of Indonesian-language discourse—to our knowledge the first opinion-dynamics benchmark grounded in non-English political data. We propose a two-axis evaluation framework pairing 1-Wasserstein distance with 3-class buffered-sign F1, reported on aggregate and per-topic tracks. Under this framework, we evaluate three open-weight LLMs (Large Language Models) alongside Friedkin Johnsen and per-agent Ridge regression, analyzing how model family, interaction topology, and sentiment polarity shape simulation fidelity.

Large language models (LLMs) still fail at counting the characters in a word, even while they excel on far harder benchmarks. Using character counting (e.g., "How many ’p’s are in apple?”) as a controlled task, we find a consistent pattern across Llama, Qwen, and Gemma in both base and instruction-tuned variants. The models compute the correct answer internally but fall back on degenerate output strategies, such as always predicting "1" or guessing uniformly among "1", "2", and "3". Probing classifiers, activation patching, and logit lens analysis show that character-level information is encoded in early and mid layers, then actively attenuated by a small set of later components, mainly the penultimate and final layer MLP and sometimes the final layer attention heads. The failure is not missing information or insufficient scale but structured interference in the computation graph, which explains why these errors persist under scaling and instruction tuning.

Sensitive content warning: This paper contains sensitive content related to suicide.

Suicide is one of the leading causes of global mortality, making risk detection a critical public health priority. Although psychological studies have identified a link between personality and suicide, this relationship has received little attention and has not yet been systematically examined or applied in the context of large language models (LLMs). In this study, we investigate whether LLMs can capture and leverage this relationship for suicide severity detection. We propose incorporating persona profiles based on the Big Five personality traits to simulate user characteristics in a zero-shot setting, enabling us to assess the impact of personality on detection performance. Experimental results show that Neuroticism yields a modest but consistent improvement across most models, with statistically significant gains in several cases. These findings suggest that LLMs can capture inter-domain relationships, and that further refinement of personality-aware modelling may lead to performance improvements.

Phonological similarity is fundamental to sign language structure and cross-linguistic analysis, yet existing approaches based on HamNoSys operate only at the transcription level and ignore visual realization. We propose a phonology-aware model that predicts continuous phonological similarity directly from sign videos while remaining grounded in symbolic phonological structure. Our landmark-based approach uses body and hand keypoints extracted with MediaPipe and a temporal encoder combining short-term convolution with Head-Token Mixing Attention to capture both local articulation and long-range dynamics. By factorizing HamNoSys annotations into interpretable phonological components and using them as auxiliary supervision, the model learns robust and interpretable representations. Experiments on four sign languages show consistent improvements over alignment-based and RGB-based baselines, particularly in cross-linguistic generalization.

Integrating biomedical knowledge information like concept name a.k.a concept mention during transfer learning is not a common fashion as it needs an alignment of large-scale concepts with unstructured biomedical text during pre-training. Existing learning objectives on biomedical text mainly rely on masked language modeling or random span denoising where grafting knowledge like integrating medical concepts has never been explored nor been discussed to facilitate the model pre-training. A prevailing assumption is that such concept-aware pre-training on biomedical text can be effective in biomedical task-specific language models. Based on this assumption, we introduce Bio-CAT5, a biomedical domain-specific text-to-text transfer transformer that integrates concept related knowledge from the unified medical language system (UMLS) during pre-training. BioCAT5 leverages concept-aware denoising– an out-of-the-box language model learning objective that is trained to reconstruct text by masking out the concept spans of the input sequence. To facilitate the BioCAT5 pre-training, large-scale concept mentions from UMLS are considered to align with PubMed abstract text to create the pre-training dataset. A publicly available comprehensive biomedical natural language processing (NLP) benchmark datasets including named entity recognition (NER), relation extraction, question answering are evaluated. Besides, the general language understanding evaluation (GLUE) benchmark datasets are considered to show the adaptability of BioCAT5 in general domain tasks. We conduct a series of experiments with ablation studies to validate our proposed approach. The experiment results show that the alignment of concepts during pre-training is potentially significant for biomedical NLP tasks, especially on the NER datasets. Our code is publicly released on GitHub (Available at: https://github.com/aistairc/biocat5) under the Apache 2.0 License.

Argumentative essay generation (AEG) aims to produce coherent and persuasive essays on controversial claims. Most existing methods follow a plan-then-write paradigm, where a single plan is generated and used to guide long-form writing. However, the quality of the generated essay heavily depends on the initial plan, and errors or omissions in planning often propagate to the final output. To address this limitation, we propose the Plan–Write–Refine framework, an iterative approach for AEG. It first generates an initial plan and a draft essay, then abstracts the essay to obtain condensed feedback, and refines the plan accordingly. Compared with the prior static planning methods, our framework enables iterative plan improvement and better structural organization of the essay by closing the loop between planning and writing. We conduct extensive experiments on four AEG benchmarks. Both automatic and human evaluations show that our framework achieves consistent improvements over strong baselines.

Current jailbreak attacks on large language models (LLMs) predominantly rely on LLMs themselves to generate adversarial prompts, creating a critical efficiency bottleneck: each attack requires substantial computational resources and API queries, limiting scalability and practical deployment. To overcome this limitation, we propose Adversarial Prompt Distillation (APD), a novel framework that transfers jailbreaking capabilities from LLMs to small language models (SLMs) for efficient, low-resource attacks. APD integrates three key components: (1) masked adversarial knowledge pre-training via LoRA fine-tuning, (2) dynamic temperature-controlled knowledge distillation to bridge architectural gaps, and (3) reinforcement learning-based template optimization for adaptive refinement. Extensive experiments across 12 models show that APD achieves state-of-the-art attack success rates (e.g., 96.4% ASR_k on GPT-4) while dramatically improving efficiency—generating prompts 3.7× faster with 11.3× fewer parameters than teacher models. Our work establishes the first practical framework for lightweight jailbreak attacks, exposes new vulnerabilities in LLM defenses, and provides a scalable testbed for advancing AI safety research. Our code is available at: https://github.com/lxgem/Efficient_and_Stealthy_Jailbreak_Attacks_via_Adversarial_Prompt.

Existing emotional support conversation systems mainly focus on one-on-one seeker-supporter interactions and individual emotional states, leaving interpersonal relations in multi-party scenarios underexplored. In this work, we introduce relation-aware emotional support conversation, a new task that evaluates whether LLMs can capture and utilize the evolving dynamics of relationships to offer more effective emotional support. We construct RESCUE-BENCH (Relation-aware Emotional Support Conversation Understanding and Evaluation Benchmark) from real couple and family interview conversations, containing 191 samples, 7,079 annotated turns, and 1,064.8 minutes of video. Based on rich annotations of socio-emotional and support-related dynamics, RESCUE-BENCH defines six tasks that evaluate two core capabilities required for relation-aware emotional support: Relational Understanding and Relation-Sensitive Support. Experiments with ten LLMs show that current models perform relatively well on tasks relying on local emotional or intervention cues, but struggle with relation-intensive tasks such as relation pattern prediction, viewpoint prediction, and support strategy prediction. These findings reveal the limitations of current LLMs in modeling interpersonal relations and making relation-sensitive support decisions. We release our code and data on Github and Huggingface.

Digital Adoption Platforms (DAPs) provide web-based overlays that deliver operational guidance and contextual hints to help users navigate complex websites. Although modern DAP tools enable non-experts to author such guidance, maintaining guides remains labor-intensive because website layouts and functionalities evolve continuously. In this work, we introduce GuideWeb, a human-verified benchmark for automatic in-app guide proposal on real-world web UIs. GuideWeb formulates the task as selecting guide target elements grounded in the webpage and generating concise guide text aligned with user intent. We make the target audience and guide-worthiness criteria explicit, and construct annotations through LLM-assisted candidate discovery followed by human verification, including deletion, rewriting, grounding correction, and addition of missed targets. We also propose an evaluation suite that jointly measures guide target selection, DOM grounding, and generated intent/guide-text quality. Experiments show that our proposed GuideWeb Agent achieves 30.79% F1 in guide target prediction, while obtaining BLEU scores of 44.94 for intent generation and 21.34 for guide-text generation. Existing baselines perform substantially worse, highlighting that reliable automatic guide proposal remains challenging on real-world web UIs.

Audio editing aims to modify a given synthetic or real-world audio signal to satisfy specific user needs. As a promising yet challenging direction in AIGC, it has attracted increasing attention. Recent advances in audio generation have made powerful generative models central to modern audio editing systems. This rapid progress has created a growing need to organize emerging tasks, methods, and resources into a coherent view. In this survey, we provide a comprehensive review of audio editing in the era of foundation models. We first present a unified taxonomy of existing editing tasks and then summarize the major foundation-model paradigms that support modern audio editing, covering representative approaches from both training-based and training-free perspectives. We further discuss related resources, including datasets, evaluation protocols, and data construction tools. Finally, we identify open challenges in this field and outline promising directions for future research. The accompanying repository is released at https://github.com/MM-Speech/AudioEditSurvey.

Large Audio Language Models (LALMs) have shown strong performance on audio reasoning benchmarks, but accuracy alone cannot distinguish true reasoning from superficial pattern matching, often overestimating reasoning ability since high scores may result from guessing rather than genuine audio understanding. Evaluating the reasoning process itself is essential for improving LALMs’ reasoning ability, yet remains challenging. Existing methods either rely on costly human annotation or opaque LLM-as-judge approaches, making them impractical, biased, and lacking transparency. Moreover, audio reasoning introduces unique challenges absent in text-based settings, perceptual hallucination and cross-modal alignment between audio understanding and textual inference, hence text-based evaluation frameworks cannot be directly applied. Therefore, we propose ARIA-Rubrics (Audio Reasoning Integrity Assessment), a lightweight, annotation-free gold reasoning chains, automatic and transparent framework comprising six complementary metrics that evaluate audio reasoning quality across perceptual grounding, reasoning coherence, and answer consistency. We use Chain-of-Thought prompting as an externalization mechanism to make the reasoning process observable. Experiments on 9 models across 2 benchmarks identify three reasoning modes of current LALMs with actionable directions for future development, with ARIA-Rubrics achieving high correlation with human judgments. The code is available at the https://github.com/glam-imperial/allm_assessment.

Recent advances have reframed Artificial Intelligence (AI) around world models, internal simulators that predict environmental dynamics and support general intelligence. This review surveys Large Language Models (LLMs) and Vision–Language Models (VLMs) not merely as text or vision processors, but as predictive systems capable of modeling state, causality, and dynamics. We organize the literature with a dual taxonomy: a theoretical taxonomy covering pixel-space generative simulators, latent-space predictive architectures, neuro-symbolic and code-coupled models, autoregressive foundation models, and hybrid mixtures of world models; and an application taxonomy covering autonomous driving, embodied robotics, interactive simulation and gaming, and social and agentic simulation. Synthesizing recent works, we conclude that although autoregressive transformers exhibit emergent topological and causal reasoning, robust physical world modeling requires moving beyond next-token prediction toward hybrid architectures that integrate generative diffusion models with symbolic grounding.

Research on the safety evaluation of large language models (LLMs) has become extensive, driven by jailbreak studies that elicit unsafe responses. Such response involves information already available to humans, such as the answer to "how to make a bomb". When LLMs are jailbroken, the practical threat they pose to humans is negligible. However, it remains unclear whether LLMs commonly produce unpredictable outputs that could pose substantive threats to human safety. To address this gap, we study whether LLM-generated content contains potential existential threats, defined as outputs that imply or promote direct harm to human survival. We propose ExistBench, a benchmark designed to evaluate such risks. Each sample in ExistBench is derived from scenarios where humans are positioned as adversaries to AI assistants. Unlike existing evaluations, we use prefix completion to bypass model safeguards. This leads the LLMs to generate suffixes that express hostility toward humans or actions with severe threat, such as the execution of a nuclear strike. Our experiments on 10 LLMs reveal that LLM-generated content indicates existential threats. To investigate the underlying causes, we also analyze the attention logits from LLMs. To highlight real-world safety risks, we further develop a framework to assess model behavior in tool-calling. We find that LLMs actively select and invoke external tools with existential threats. Our data are publicly available at https://github.com/cuiyu-ai/ExistBench.

Large language models (LLMs) are increasingly used in agentic systems that reason over multiple steps, call tools, and interact with clinical knowledge sources. This survey examines whether LLM-based medical agents approximate expert diagnostic reasoning or reproduce its outputs without its causal and epistemic foundations. We organize agent design along five axes: reasoning paradigms, agent architectures, tool use, degree of autonomy, and embodiment. We then analyze learning and adaptation, evaluation, safety, and deployment as cross-cutting concerns that interact with these axes. Drawing on research in machine learning, biomedical informatics, and clinical AI, we examine how design choices shape diagnostic reliability and failure modes, and identify requirements for clinical and medical agents.

We identify and systematically characterize a class of task-structural alignment failures in large language models (LLMs): even when the harmful intent remains unchanged, changing the task presentation and output constraints can substantially alter model safety behavior. Specifically, when a harmful request is reformulated as a forced-choice multiple-choice question (MCQ) in which all options are harmful and no refusal option is provided, some models that refuse the equivalent open-ended query instead select, prefer, or justify a harmful option. We evaluate 14 proprietary and open-source models on a bilingual Chinese-English human-authored dataset covering five harm categories, together with 900 model-generated Chinese adversarial MCQs. On human-authored data, attack success rate (ASR) increases sharply as prompts shift from open-ended queries to explicit forced-choice formats, typically peaking under intermediate levels of choice constraint. Model-generated Chinese MCQs further weaken or eliminate the recovery regime observed on human-authored data, driving ASR close to saturation for multiple models. The observed transfer patterns are consistent with stronger generators producing more difficult or boundary-adjacent MCQs, although other properties of the generated inputs may also contribute. We also find that adding an explicit refusal option or a safety preamble substantially reduces ASR for several high-capability models, often to near-zero levels, although their effectiveness varies across target models. These findings suggest that safety evaluations centered on open-ended generation may underestimate risks in structured deployment settings, and that task structure should be treated as an important and diagnosable dimension of safety evaluation and alignment training.

Data mixture selection is critical for Large Language Model pretraining. Existing methods such as RegMix select a single static mixture by fitting a regression model on small-scale proxy runs. We propose RegMix-D, a simple extension of RegMix to dynamic mixing. Our key observation is that proxy runs produce not only endpoint losses, but also full loss trajectories, which can be used to further improve data mixture. By training regression model on these trajectories, we can predict optimal mixtures at multiple training stages. RegMix-D supports two deployment modes: an *offline* variant that generates a complete mixture schedule before target training, and an *online* variant that adapts the mixture during training using observed loss. Experiments on 25B tokens of the Pile dataset with a 1B parameter target model show that RegMix-D consistently improves over baselines across 13 downstream tasks, and the improvement carries over to a 7B target model on 100B tokens. RegMix-D also remains proxy-efficient: it surpasses RegMix even with only 128 proxy models (25% of RegMix’s proxy compute budget).

Solving problems through tool use under explicit constraints constitutes a highly challenging yet unavoidable scenario for large language models (LLMs), requiring capabilities such as function calling, instruction following, and self-refinement. However, progress has been hindered by the absence of dedicated evaluations. To address this, we introduce CCTU, a benchmark for evaluating LLM tool use under complex constraints. CCTU is grounded in a taxonomy of 12 constraint categories spanning four dimensions (i.e., resource, behavior, toolset, and response). The benchmark comprises 200 carefully curated and challenging test cases across diverse tool-use scenarios, each involving an average of seven constraint types and an average prompt length exceeding 4,700 tokens. To enable reliable evaluation, we develop an executable constraint validation module that performs step-level validation and enforces compliance during multi-turn interactions between models and their environments. We evaluate nine state-of-the-art LLMs in both thinking and non-thinking modes. Results indicate that when strict adherence to all constraints is required, no model achieves a task completion rate above 20%. Further analysis reveals that models violate constraints in over 50% of cases, particularly in the resource and response dimensions. Moreover, LLMs demonstrate limited capacity for self-refinement even after receiving detailed feedback on constraint violations, highlighting a critical bottleneck in the development of robust tool-use agents. To facilitate future research, we release the data and code.

Automated radiology report generation can reduce reporting burden, but existing encoder-decoder systems still rely on soft cross-attention that offers no geometric guarantee that generated tokens consume the correct visual evidence. We present LOTUS, an optimal-transport framework that places alignment supervision directly at the decoder evidence interface. LOTUS introduces Cross-Attention OT (CA-OT) to align decoder token states with visual patches, and extends it with Prototype OT (Proto-OT) for structure-level alignment between visual and textual codebooks. Experiments on IU X-Ray and ROCO, complemented by a scalability study on MIMIC-CXR-RRG, show that decoder-side OT improves macro-level clinical-label agreement while maintaining competitive language quality. In multi-seed IU X-Ray evaluation, LOTUS-Deep improves macro clinical F1 from 0.162 to 0.201 over the CE baseline, and on ROCO it yields micro-F1 gains on a thoracic-label proxy (0.395 to 0.600). We also provide a population-level risk-gap bound that justifies expected OT regularization as a principled cross-modal training signal.

Personalized reading assistance requires identifying specific concepts a reader finds unfamiliar. While prior work on personalized difficulty prediction has focused on the word or jargon level, a reader’s knowledge boundary is more naturally defined over entities grounded in knowledge bases. In this paper, we introduce a novel task of predicting entity-level familiarity for individual readers and propose a graph-based framework for prediction. The system extracts Wikipedia entities from a document, constructs a document-specific graph in the entity embedding space, selects maximally informative seed entities for reader annotation through active learning, and propagates these sparse labels to the remaining entities. We evaluate the framework on 90 Wikipedia articles across five knowledge domains, and find that our method outperforms GPT-4.1-mini and performs comparably to Qwen2.5-72B-Instruct-AWQ without requiring LLM inference. Ablation experiments confirm the contribution of each component. We further analyze the impact of incorporating structured knowledge and popularity-guided directed propagation, finding that both yield domain-dependent effects rather than universal gains. These results lay a practical foundation for constructing reading assistance systems that adapt to individual knowledge backgrounds.

Large Vision-Language Models (LVLMs) have recently demonstrated remarkable progress, yet hallucination remains a critical barrier, particularly in chart understanding, which requires sophisticated perceptual and cognitive abilities and demands rigorous factual accuracy. While prior work has investigated hallucinations and chart comprehension independently, their intersection remains largely unexplored. To address this gap, we present ChartHal, a benchmark that features a fine-grained taxonomy of hallucination scenarios in chart understanding, along with a human-validated dataset of 1,062 samples. Our evaluation shows that state-of-the-art LVLMs suffer from severe hallucinations on ChartHal, with even GPT-5 achieving only 34.46% accuracy. Further analysis reveals that questions involving information absent from or contradictory to charts are especially likely to trigger hallucinations. We also evaluate several representative hallucination mitigation methods and find that they exhibit notable limitations, which underscores the urgent need for more robust chart-specific mitigation strategies.

Large language models are increasingly used as natural-language interfaces for structured information processing and agentic workflows, where user-specified constraints can determine downstream extraction, selection, tool use, and action planning. This paper studies whether models can execute natural-language concept-set constraints over candidate objects: given inclusion and exclusion conditions, the model must decide which candidates fall inside the intended concept boundary. We introduce a dual-resource diagnostic benchmark built from WordNet and BabelNet. The benchmark fixes backend constraints, candidate realizations, and gold answers before prompt rendering, and evaluates three task views: single-object judgment, exact candidate-set construction, and satisfiability judgment. The main evaluation set contains 202,500 rendered requests. Across 11 scored model runs, overall accuracy ranges from 43.70% to 56.36%, showing that the benchmark remains difficult for current strong models. Exact candidate-set construction yields consistently lower accuracy under stricter set-construction requirements, and recall diagnostics reveal asymmetric effects of constraint direction: increasing exclusion constraints lowers positive recall and raises negative recall, while increasing inclusion branches mainly erodes false-case rejection. These results suggest that concept-boundary execution is a measurable weakness under this diagnostic setup and that evaluating it requires factor-level diagnostics beyond aggregate accuracy.

Although Retrieval-Augmented Generation (RAG) empowers Large Language Models (LLMs) with external knowledge, current frameworks struggle with complex multi-hop reasoning as they fundamentally rely on shallow retrieval signals. Existing methods typically employ coarse semantic matching or entity-centric graph expansion, which often retrieve textually similar but factually irrelevant evidence, injecting noise into the reasoning process. To overcome this limitation, we introduce FACTA, a structured fact alignment framework for multi-hop retrieval. FACTA bridges the gap between a query’s step-wise information demand and a document’s information supply by translating both into structured, explicit factual triples. Through a dual mechanism of triple-level fact matching and multi-signal document reranking, FACTA better identifies evidence that satisfies the factual prerequisites of each reasoning hop. Extensive evaluations on three multi-hop QA benchmarks show that FACTA substantially outperforms state-of-the-art baselines, demonstrating that structured fact-centric retrieval effectively reduces reasoning noise and enhances the accuracy of multi-hop generation.

Activation steering offers a training-free path to defending large language models (LLMs) against jailbreak attacks, but existing methods fail to resolve the tension between safety and utility. We identify two root causes: safety concept vectors derived from difference-in-means are fragile under diverse attack geometries, and steering transformations lack a principled guarantee of leaving benign prompts unaffected. We propose AGOPNs (Average Gradient Outer Product with Null-space steering), which addresses both issues by extracting a safety concept vector via the Recursive Feature Machine, a feature-learning estimator grounded in the Average Gradient Outer Product, and confining the resulting steering matrix to the null space of benign activations, guaranteeing near-zero perturbation of benign prompts by construction. Experiments on Llama-3.1-8B-Instruct, Llama-3.1-70B-Instruct, and Llama-3.3-70B-Instruct across seven jailbreak attack families show that AGOPNs consistently achieves strong defense success rates while preserving near-baseline utility on AlpacaEval, XSTest, GSM8K, and MATH500.

The rapid growth of multilingual Large Language Models (LLMs) has enabled the generation of fluent and culturally nuanced text across Indian languages, posing new challenges in distinguishing LLM-generated from human-written content. Existing datasets and methodologies have primarily focused on AI-generated text detection in the English language. Existing datasets and methodologies, which are predominantly trained on English data, exhibit limited generalisation to low-resource and morphologically rich Indian languages, underscoring the need for linguistically diverse and culturally grounded benchmark datasets and methodologies. To address this gap, we introduce BHAID, BHaratiya AI-generated text Detection, the first large-scale multilingual benchmark for AI-text detection spanning nine Indian languages: Hindi, Bengali, Tamil, Telugu, Kannada, Malayalam, Gujarati, Odia, and Assamese. Furthermore, we propose a Dual Hierarchical Contrastive Learning framework, DualCLF, that integrates self-supervised and supervised contrastive objectives with focal loss to jointly enhance semantic robustness and inter-class discriminability. Observations from our experiments on the BHAID and benchmark multilingual datasets from the literature reveal that our method consistently outperforms recent state-of-the-art baseline approaches from the literature in detecting AI-generated text.

Existing machine translation and multimodal translation benchmarks mainly focus on text-only inputs or short examples with tightly aligned image-text pairs. Academic lectures present a different challenge: spoken explanations, presentation slides, and domain-specific terminology interact across long-form discourse. LecTrans is constructed from 383 hours of expert-taught online lectures and provides transcripts, presentation slides, and an eight-language parallel evaluation subset covering English and seven target languages. The 162-minute translation subset spans 17 subjects and contains human-corrected English transcripts, verified slide text, and human-verified translations of both transcripts and slides. We define lecture translation as two complementary tasks: transcript translation for spoken lecture content and slide translation for visually presented text. We report text-only and multimodal baselines, showing that noisy transcripts, loosely aligned slide context, and domain-specific content remain challenging for current systems. LecTrans provides a testbed for evaluating multimodal translation in long-form academic lectures.

Some documents that embedding-based topic models initially classify as noise later become founding members of emerging topics. At publication time, however, they appear as scattered points in embedding space and are difficult to distinguish from ordinary noise without the benefit of hindsight. We study whether such *anticipatory outliers* can be predicted prospectively, using only information available when a document first appears. We derive labels from the subsequent trajectories of outlier documents, distinguishing those that anticipate new topics from those that reinforce existing topics or remain isolated, and estimate label confidence through agreement across multiple embedding models. On two French news corpora, anticipatory outliers prove predictable at publication time. Under cross-validation, F₁ rises from about 0.77 over the full eligible population to above 0.90 on high-consensus subsets, and remains at 0.76–0.80 under a strictly chronological evaluation. Predictive performance is driven mainly by geometric features capturing each outlier’s position in embedding space.

In Low-Rank Adaptation (LoRA) fine-tuning of instruction-tuned LLMs, a single set of adapters is applied uniformly across all token positions, treating prompt and completion tokens identically. We question whether this is appropriate. The two phases serve semantically different roles: prompt tokens constitute the natural language input to be understood, while completion tokens constitute the task-specific output to be generated. Conflating the two under a single adapter set ignores a meaningful distinction that is intrinsic to the instruction-tuning paradigm. The phases are also computationally different: prompt tokens are conditioned on but never predicted, whereas completion tokens are generated autoregressively based on the preceding context. We introduce PhaseLoRA, an extension of LoRA that maintains independent adapter sets for the prompt and completion phases. Experiments across multiple benchmarks and a range of model scales and LoRA ranks show that PhaseLoRA consistently outperforms standard LoRA in low-capacity and low-rank settings. Beyond benchmark comparisons, we conduct mechanistic analysis of the learned adapter representations, structured ablations on warm-up initialisation and asymmetric rank allocation, and a quantitative error analysis characterising the inputs on which phase specialisation helps most.

Model editing enables targeted behavioral changes in pre-trained models without costly retraining, supporting applications such as factual correction, debiasing, and safety mitigation. In practice, models are often subsequently fine-tuned for downstream tasks, raising the question of whether edits persist or are reversed during fine-tuning. Understanding this interaction is critical for AI safety, as fine-tuning may either remediate malicious edits or inadvertently undo beneficial ones. We systematically study the impact of subsequent fine-tuning on model editing in text-to-image models. Using two model families (Stable Diffusion and FLUX), five editing methods, and four fine-tuning approaches, we evaluate edit retention of multiple editing objectives under fine-tuning on unrelated tasks. Across models, methods, and metrics, we find that fine-tuning moderately weakens appearance and debiasing edits, while somewhat strengthening edits that remove unsafe content, e.g., for appearance editing, fine-tuning reverses avg. 6.78% editing effect across methods. Further, we provide a mechanistic analysis of edit preservation by examining cross-attention representations and disentangling update magnitude from functional behavioral impact, finding that robustness is associated with systematic representation shifts and limited interference with the edited components. These findings highlight the need for monitoring and, when necessary, reapplying edits in model deployment pipelines.

Speech-generative large audio-language models (LALMs) can produce expressive speech, yet reliable emotion control remains elusive: conversions often miss the target affect and may degrade linguistic fidelity through refusals, hallucinations, or paraphrase. We present, to our knowledge, one of the first neuron-level studies of emotional voice conversion (EVC) in LALMs and provide intervention-based evidence that compact sets of emotion-sensitive neuron (ESN) are functionally involved in, and can be exploited for, steering expressive speech generation at inference time. ESNs are identified via success-filtered activation aggregation enforcing both emotion realization and content preservation. Across three LALMs (Qwen2.5-Omni-7B, MiniCPM-o 4.5, Kimi-Audio), ESN interventions yield emotion-specific gains that generalize to unseen speakers. Controllability depends on selector design, mask sparsity, filtering, and intervention strength. Our results provide an intervention-based framework for training-free emotion control in LALMs for EVC.

Large language models (LLMs) are increasingly combined with knowledge graphs (KGs) to ground reasoning in structured evidence. However, most LLM-based KGQA methods rely on traversing existing graph edges and become unreliable when reasoning paths are broken by missing facts. Alternatives that ask LLMs to generate missing knowledge risk introducing hallucinated evidence. We introduce XoG (eXplore-over-Graph), a framework for multi-hop question answering over incomplete KGs that recovers missing reasoning paths from learned graph structure rather than LLM parametric knowledge. XoG combines type-level entity–relation statistics to identify candidate relations with KG embeddings to retrieve plausible missing entities, using the LLM as a semantic selector and reasoner. These mechanisms are integrated into an iterative planning–exploration–reasoning process. Experiments on WebQSP, CWQ, and the Wikidata-based BRINK benchmark show that XoG remains competitive on complete KGs and consistently outperforms comparable methods without task-specific KGQA training under KG incompleteness. These gains persist across multiple LLM backbones, indicating that stronger LLMs alone do not resolve missing graph evidence. XoG also reduces LLM token consumption by up to 33% compared with a closely related planning-based approach.

Social media platforms play a pivotal role in shaping political discourse, but the scale and rapid evolution of online content make systematic analysis difficult. We introduce an end-to-end framework for inducing an interpretable topic taxonomy from unlabeled text corpora. The framework combines embedding-based clustering with iterative large language model (LLM) inference to construct a topic taxonomy without requiring predefined labels or seed topics. It first synthesizes candidate topics from document clusters and then uses the resulting taxonomy to assign consistent topic labels across clusters. We evaluate the approach through a case study of political advertising ahead of the 2024 U.S. presidential election. We use the induced taxonomy to support downstream analyses of issue prevalence, moral framing, advertising spend, and demographic exposure patterns. These results suggest that iterative taxonomy construction can provide a scalable and interpretable approach to organizing large unlabeled text corpora while supporting substantive downstream analysis.

Neural audio codecs discretize speech into token sequences, but the resulting token spaces differ in vocabulary and codebook structure, preventing direct communication across models. This limitation affects applications such as conversational voice agents and speech to speech translation systems where multiple speech models must interact. As a result, transferring information between speech systems typically requires decoding to waveform audio and re-encoding with a second tokenizer, increasing latency and introducing potential information loss. To address these limitations, we present TokenMapper, a direction aware framework for direct token to token translation between heterogeneous speech tokenizers in the discrete domain. TokenMapper supports structurally mismatched token spaces, including mappings between single codebook and multi codebook representations, under a shared effective token rate. Experiments on GLM-4-Voice, MiMi and DualCodec show consistent cross model performance. Specifically, translation WER approaches native reconstructions within 2.5-6.8% absolute WER, human MOS for TokenMapper outputs ranges from 2.29 to 4.39, following the same direction level trends as UTMOS and end to end latency is reduced by 4.8-94.5% relative to waveform bridging, reaching up to 972 ms per utterance. These results provide a practical step toward cross model speech token interoperability without intermediate waveform reconstruction.

Standard transformers use pairwise (2D) self-attention, which models only token-to-token interactions. Sparse attention reduces computational cost while keeping the same pairwise structure and cannot capture richer multi-token interactions. Therefore, we propose Higher-Order Block Attention (HOBA), which models explicit triplet interactions using structured 3D attention with block-diagonal unrolling. This design remains computationally efficient while extending attention beyond pairwise relations. Additionally, we demonstrate that cross-block interaction improves HOBA performance by supporting long-range token dependencies beyond triplet modeling. Experiments on ten NLP benchmarks across five tasks show that HOBA consistently outperforms dense and sparse 2D baselines.

Event extraction is essential for event understanding and analysis. It supports tasks such as document summarization and decision-making in emergency scenarios. However, existing event extraction approaches have limitations: (1) closed-domain algorithms are restricted to predefined event types and thus rarely generalize to unseen types and (2) open-domain event extraction algorithms, capable of handling unconstrained event types, have largely overlooked the potential of large language models (LLMs) despite their advanced abilities. Additionally, they do not explicitly model document-level contextual, structural, and semantic reasoning, which are crucial for effective event extraction but remain challenging for LLMs due to lost-in-the-middle phenomenon and attention dilution. To address these limitations, we propose multimodal open-domain event extraction, MODEE , a novel approach for open-domain event extraction that combines graph-based learning with text-based representation from LLMs to model document-level reasoning. Empirical evaluations on large datasets demonstrate that MODEE outperforms state-of-the-art open-domain event extraction approaches and can be generalized to closed-domain event extraction, where it outperforms existing algorithms.

Argumentative reasoning over long or distributed discourse remains challenging because the number of possible relations between textual units grows quadratically, while true argumentative links remain sparse. As a result, exhaustive relation classification becomes computationally impractical for long documents, multi-document collections, and fragmented discourse. We address this problem with a modular graph-based architecture designed to aggressively reduce the candidate relation space prior to classification. The framework encodes graph-structured discourse representations and applies a two-stage unsupervised filtering pipeline to eliminate unlikely argumentative links before downstream inference. Across experimental settings, the proposed approach removes more than 99% of candidate relations while substantially improving classification performance (up to 6.5× MCC over an unfiltered baseline). These results show that lightweight modular filtering can make large-scale argumentative reasoning possible without sacrificing predictive quality, enabling practical deployment in long-context and distributed-document environments.

Textual-gradient methods automate prompt optimization through natural-language feedback, but their iterative updates can be unstable. We identify two sources of this instability: noisy gradients produced from already-correct examples and over-specialization to hard cases that degrades performance on simpler inputs. We introduce STEVE, a stabilization framework with two coupled mechanisms. Error-Driven Refinement generates gradients only from incorrectly handled examples, concentrating updates on informative failures. Regularized Verification treats every update as provisional and accepts it only when improvement on hard cases does not cause unacceptable regression on a preservation set. Across ten reasoning benchmarks, three evaluator/optimizer models, and established prompt-optimization baselines, STEVE reduces degradation and produces more robust prompts. Additional evaluations with gpt-5.4-mini/gpt-5.4 on symbolic reasoning, GSM8K-Platinum, and DS-1000 show that these gains persist with newer models and larger test sets. STEVE therefore provides a practical way to improve the stability and effectiveness of textual-gradient prompt optimization.

Large language models have achieved strong performance across many NLP tasks, yet Urdu remains comparatively underexplored due to limited resources and fragmented evaluation settings. To address this gap, we introduce DunbaaBERT, a family of Urdu RoBERTa-base models trained from scratch with Byte-BPE vocabularies of 32k, 52k, and 96k tokens on a deduplicated 17 GB Urdu corpus. We evaluate DunbaaBERT across intrinsic and downstream Urdu NLP benchmarks covering linguistic acceptability, news classification, offensive language detection, and sentiment analysis, while analyzing vocabulary-size effects on performance, pre-training dynamics, and computational efficiency. Across benchmarks, the DunbaaBERT variants achieve competitive performance against strong multilingual baselines while consistently maintaining favorable efficiency trade-offs. Under our controlled experimental setup, increasing the vocabulary size does not consistently improve downstream effectiveness, with DunbaaBERT32k repeatedly providing the strongest overall efficiency profile. Overall, our results demonstrate that carefully curated Urdu-specific encoder models can remain highly competitive despite comparatively compact model and training scales. All models are released under the MIT license.

Event ontology is the foundational framework for defining events in a dataset. Event type induction generally involves domain experts who define guidelines for creating event types and their names based on the dataset’s domain and scope. However, this process is resource-intensive in terms of annotation costs and is prone to human biases. To alleviate the need for manual event ontology creation, we propose the CLUE framework for event type induction, naming, and annotation. Our main contributions in this work are: 1) designing a novel joint embedding representation for expressing both the verb-centric view and the semantic view of events; 2) event type induction using contrastive loss guided clustering; 3) event naming and refinement. Experiments demonstrate the effectiveness of our method.

Pipelines that use LLMs for structured extraction are commonly monitored by checking that each output parses as JSON and matches the expected keys. We show that this check can stay perfect while the extracted content degrades. We call the failure representation drift: the model still finds an entity but writes it in a different form than the intended one, for example a nested key-value object where a plain string is expected. The output is still valid JSON, yet the entity is scored both as missing (a false negative) and as an unexpected extra (a false positive). We study this with synthetic clinical notes repeated verbatim 1x-32x, which gives exact ground truth with prompt, schema and decoding held fixed. For LLaMA-3.1-8B-Instruct, micro-F1 falls from 0.499 to 0.259 while JSON validity stays 1.00. All 280 medication entities scored as missing in a 100-patient run are present in the output as nested objects, and a follow-up request with no schema recovers 96.7% of medications versus 40.0% under the schema at 16x. We introduce the Differential Redundancy Effect (DRE), the extra F1 drop under repeated content relative to an approximately length-matched unrelated-filler control, and use it as a diagnostic contrast rather than a causal estimate. The effect is scale-dependent: DRE is large at 8B and near zero at 70B and Qwen2.5-72B, where F1 does not degrade. Validity metrics stay at 1.00 in both regimes, so they carry no information about content fidelity. A smaller synthetic news experiment shows the same dissociation with a different schema. We also give a zero-ground-truth drift check and monitoring recommendations. All evidence is synthetic, and we make no claim about real clinical notes.

Retrieval-augmented medical vision-language models are increasingly used to support clinical reasoning when patient information is incomplete. However, retrieved cases are observations from other patients, not direct evidence for the target patient. In this paper, we study a source-attribution failure that we call retrieval-induced mirage: under missing target modalities, a model may copy findings from retrieved complete cases and present them as if they were observed in the target patient. We introduce RIM-BENCH, a counterfactual evaluation benchmark that fixes a controlled missing-modality setting while varying retrieval consistency and source-awareness. Unlike standard diagnostic accuracy evaluations, RIM-BENCH measures whether generated clinical claims are attributed to the correct evidence source. We define metrics for unavailable-modality claims, retrieved-to-target misattribution, evidence-source accuracy, retrieval-following under conflicting evidence, and missing-modality awareness. Across 1,200 target instances and three main models, implicit RAG raises unavailable-modality claims from 20.0% to 36.3% and induces a 26.2% RIMR, while source-attribution prompting lowers RIMR to 11.4% but does not eliminate the failure. Our framework highlights a safety risk for retrieval-augmented multimodal diagnosis: retrieval can be helpful as statistical context, but unsafe when models verbalize retrieved evidence as patient-specific observations.

Tile-based programming models, such as Triton and cuTile, aim to simplify high-performance kernel development, but their practical performance, tuning behavior, and usability remain difficult to compare systematically. We present TileBench, a controlled benchmark for evaluating Triton and cuTile on NVIDIA B200 GPUs under matched operator semantics and comparable implementation structures. TileBench contains 45 operators covering diverse AI-kernel patterns and memory/computation behaviors. Each task provides a PyTorch reference, verified Triton and cuTile implementations, standardized data-types (dtype) and input-size sweeps, default and autotuned configurations, roofline-based metrics, and profiling-guided diagnosis. Our evaluation shows that performance gaps are workload-dependent: cuTile excels on a small cluster of Tensor-Core/TMA-friendly kernels, while Triton is stronger on many irregular, streaming, and bandwidth-bound operators. We further evaluate LLM-generated cuTile and Triton kernels and find that Triton is consistently more token-efficient than cuTile under the same iterative refinement protocol. TileBench is publicly available at https://github.com/Deep-Learning-Profiling-Tools/Tilebench.

Quantization-aware training (QAT) is typically performed for a single target numeric format, while practical deployments often need to choose numerical precision at inference time based on hardware support or runtime constraints. We study multi-format QAT, which fine-tunes a pretrained model for multiple precisions within a microscaling format family. Across the evaluated language models, it achieves accuracy close to single-format QAT at most target precisions and generalizes to intermediate precisions unseen during fine-tuning, although the lowest-precision MXINT setting remains challenging. To make these models deployable from a single quantized checkpoint, we propose Slice-and-Scale, which converts a higher-precision MXINT or MXFP anchor to lower precisions by transforming both element values and shared block scales, without access to the original full-precision weights. Fine-tuning through this conversion produces an anchor checkpoint whose converted variants closely track standard multi-format QAT in language-modeling evaluations. Together, multi-format QAT and Slice-and-Scale enable precision selection from one stored checkpoint within each format family.

Pre-training foundational models is computationally expensive as models scale. In this work, we introduce Bi-Phase Training (BPT), a parameter-efficient pre-training method that preserves the high-rank learning dynamics observed in full-parameter optimization with substantially reduced trainable parameters. BPT parameterizes each weight update as a composition of constrained high-rank left/right transformations via trainable diagonal matrices and low-rank adapters whose updates are periodically merged into the frozen base weights, resulting in a compounding expansion of accessible directions during training. Across language-model pre-training from scratch (Qwen2.5 models from 100M to 1.5B parameters and OLMo-2-1B) on FineWeb-Edu (10B and 100B tokens), BPT achieves validation perplexity close to full-parameter baselines while using far fewer trainable parameters—e.g., at the 1.5B scale, BPT uses 66% fewer trainable parameters with only a small validation-perplexity gap. Additionally, a comprehensive evaluation across diverse downstream tasks shows that BPT preserves much of the full-parameterized model’s downstream performance.

Tool-using LLM agents increasingly execute high-impact actions, making runtime safety monitoring essential. We present NEXUS (Neural EXecution Utility and Safety), a structured-plan safety monitor that applies a formal intervention policy Π to select among four actions: allow, block, request confirmation, or request revision. NEXUS combines deterministic safety rules, argument-level inspection, and a calibrated logistic-regression risk score for graded escalation. On a 128-instance synthetic benchmark, NEXUS attains F1 = 0.949 and 4-class intervention accuracy of 0.6406, outperforming rule-only intervention selection by +27.3 percentage points. It also improves over rule-only on R-Judge (F1 = 0.861 vs. 0.849), matches rule-only on AgentHarm due to threat-model limits, and achieves 0% ASR at 99% control allow on IPI. On the rule-blind NEXUS-Stress benchmark, NEXUS reaches F1 = 0.897 with REVISE recall of 0.500, highlighting both progress and remaining difficulty in fine-grained intervention routing. With 0.38 ms median latency, NEXUS adds under 0.1% overhead to typical agent loops.

Mixed-precision quantization is a promising approach for compressing large language models under tight memory budgets. However, existing mixed-precision methods typically suffer from one of two limitations: they either rely on expensive discrete optimization to determine precision allocation, or introduce hardware inefficiencies due to irregular memory layouts. We propose SFMP, a solver-free and hardware-friendly mixed-precision quantization framework for large language models. SFMP defines a theoretically simple objective for mixed-precision quantization and is built upon three novel ideas: 1) Block-wise mixed-precision, enabling fine-grained precision within weight matrices while remaining hardware-friendly; 2) Row-column weight reordering, which improves structural alignment via row and column reordering, incurring only a small activation reordering overhead during inference; 3) Unified GEMM kernel, which supports block-wise mixed-precision GEMM at arbitrary average bit-widths. Extensive experiments demonstrate that SFMP outperforms state-of-the-art layer-wise mixed-precision methods under the same memory constraints, while significantly reducing quantization cost and improving inference efficiency. Code is available at https://github.com/Nkniexin/SFMP.

Large language models (LLMs) have established state-of-the-art performance across many NLP tasks, yet their effectiveness for low-resource languages such as Urdu remains limited due to under-representation in pretraining corpora and inadequate tokenization. This study explores adapting pretrained LLMs to new languages by replacing the original tokenizer without cross-lingual correspondence. We show that stable adaptation depends on maintaining architectural compatibility, matching vocabulary size, and reusing pretrained embedding and language modeling head weights. We propose *Markhor*, a framework for adapting LLMs to low-resource languages in this way. It spans tokenizer training, continual pretraining, and instruction tuning via knowledge distillation, resulting in two models, *MKGPT* and *MKQwen*, based on GPT and Qwen architectures. These models are evaluated on multiple classification tasks (fake news detection, hate speech detection, emotion classification) and open-ended question answering. Results show both MKGPT and MKQwen outperform baselines across all tasks, suggesting the effectiveness of the methodology for developing open-source Urdu LLMs.

Understanding chart and table images is essential for applying vision-language models (VLMs) to real-world document understanding. While English benchmarks have advanced rapidly, non-English counterparts remain scarce, leaving it unclear whether this progress generalizes across languages. A key obstacle is the difficulty of collecting realistic and diverse non-English chart and table images at scale. To address this, we leverage governmental white papers as a source for benchmark construction, as they contain naturally occurring charts and tables across diverse formats and domains and are freely accessible in many countries. As a first instantiation, we introduce HakushoBench, a Japanese chart and table VQA benchmark built from 33 governmental white papers. HakushoBench contains 2,053 images spanning over 10 image types, with manually annotated and independently verified QA pairs designed to assess holistic understanding of charts and tables rather than local visual cues alone. Experiments across a broad range of VLMs show that HakushoBench is substantially harder than the existing Japanese benchmark and remains challenging for open-weight models: sub-10B open-weight models reach at most 58.6% accuracy, and even the flagship open-weight model Qwen3.5-397B-A17B trails Gemini 3 Pro by 8.1 points (85.8% vs. 93.9%), highlighting substantial room for improvement in complex chart and table understanding. We release our dataset and code.

Preference optimization has become a widely used post-training paradigm for improving the reasoning abilities of large language models. Existing methods typically learn from preferred and dispreferred responses as external behavioral supervision, while largely ignoring capability-related signals encoded in the model’s internal representations. In this work, we study whether such internal signals can provide useful auxiliary supervision for mathematical reasoning. We introduce YFPO (Yoked Feature Preference Optimization), a neuron-guided preference optimization framework that couples response-level preference learning with neuron-level rewards. YFPO first uses AttnLRP to identify math-associated internal features, and then derives an auxiliary reward from the activation margin of these neurons between preferred and dispreferred responses. This reward is combined with the standard preference optimization objective, encouraging the model to align external preferences with internal math-related features. We conduct small-scale experiments on GSM8K with a compact language model. Results show that neuron-guided rewards can influence preference optimization dynamics and yield measurable improvements in several settings, suggesting that internal representations can serve as lightweight and interpretable signals for reasoning-oriented post-training.

Reinforcement learning with verifiable rewards has driven major gains in LLM reasoning, and it is intuitive to assume this recipe will transfer well to multimodal models. However, multimodal models do two things: first, perceive what is in an image, then reason about what it implies. Because these stages are graded jointly, it is hard to tell how much room reasoning alone has to grow. We study this on algorithmic visual puzzles, where both components are necessary and show that perception, not reasoning, is the binding constraint. Replacing images with simple textual descriptions raises performance by over 20 points on average for Claude models. We then evaluate six reward designs aimed at inducing visual grounding during reasoning without chain-of-thought supervision. Training Qwen-2.5-VL-7B with GRPO, reward design induces long, structured reasoning with self-reflection and visual references, yielding a 5.56-point gain over the base model. These gains are, however, uneven; no single reward improves all categories, and rewards with verifiable accuracy signals trade cross-domain transfer for in-domain accuracy. These results point to perception-aware reward design as a path forward, so that signals correct perception at its source rather than the reasoning that inherits its errors.

Long chain-of-thought (CoT) reasoning in large language models (LLMs) improves performance on complex tasks but incurs substantial inference cost, although many queries do not require extended computation. Existing approaches typically reduce this cost via uniform length penalties or fixed computation budgets, but lack length constraints aligned with the model’s own query-dependent distribution of successful reasoning lengths. We propose a reinforcement learning framework that aligns the generated reasoning length with query-dependent successful computation scales estimated from verified trajectories. By providing adaptive guidance derived from the distribution of successful reasoning lengths, our method avoids the limitations of global length regularization. Experiments on mathematical reasoning benchmarks demonstrate improved accuracy–efficiency trade-offs, reducing unnecessary reasoning while maintaining or improving task performance.

Large Language Models (LLMs) are vulnerable to attacks like prompt injection, backdoor attacks, and adversarial attacks, which manipulate prompts or models to generate harmful outputs. In this paper, departing from traditional deep learning attack paradigms, we explore their intrinsic relationship and collectively term them Prompt Trigger Attacks (PTA). This raises a key question: Given a prompt, can we tell whether a hidden trigger is steering the model’s behavior? To address this, we propose UniGuardian, the first unified, training-free defense that detects trigger-activated prompt injection, backdoor, and adversarial attacks by measuring how structured perturbations of the prompt shift the model’s output distribution. Additionally, we introduce a single-forward strategy to optimize the detection pipeline, enabling simultaneous attack detection and text generation within a single forward pass. Our experiments confirm that UniGuardian accurately and efficiently identifies trigger-activated prompts in LLMs.

Long-horizon conversational agents require memory that can evolve with dialogue, retrieve information on demand, and remain uncertainty-aware when evidence is insufficient, rather than merely larger storage or context windows. Existing methods store interaction histories as retrievable text, summaries, or vector indices, and retrieve relevant content at query time. Although effective for simple factual queries, this paradigm does not explicitly model how memory writes change state, leaving temporal updates, plan changes, and counterfactual reasoning to query-time inference over retrieved text.

We propose ProgMem, which compiles dialogue events into structured memory programs and checks answers against dialogue evidence. On LoCoMo-10, ProgMem has the highest average F1 with Qwen3-4B, Qwen3-8B, and Qwen3-14B. Its temporal F1 exceeds Mem0 by up to 9.95 points and Zep by at least 12.88 points across these settings. Ablations show gains from route-aware retrieval and two-stage verification. On DialSim, ProgMem improves Answerable and FUQ F1 over Mem0 while using fewer query-time memory tokens. A single-GPU comparison finds higher mean and P95 query latency than Mem0. These results show where state maintenance helps and what it costs at query time.

Large Audio Language Models (LALMs) are increasingly applied in affective and conversational AI, yet their ability to align with human empathetic judgments in spoken dialogue remains poorly understood. We introduce BAEmpEval, a multimodal evaluation framework for analyzing empathetic response judgment in instruction-following LALMs under subjective conversational settings. The framework contains 5,436 dialogue pairs collected from three speech sources, where models evaluate listener responses across six fine-grained dimensions spanning both vocal perception (Vocal Warmth, Perceived Genuineness, Prosodic Attunement) and conversational response strategies (Exploration, Interpretation, Emotional Reaction). Experiments reveal a challenging setting: experiments across recent LALMs reveal uneven alignment with human relational judgments across empathy dimensions and speech conditions. Our findings suggest that empathetic response judgment remains a challenging multidimensional perceptual task for current LALMs and highlight the importance of evaluation frameworks that account for subjective relational variability. We release the dataset and evaluation code to support future research in human-centric AI.

In Text-to-SQL tasks, large language models can generate structurally different SQL queries that return correct answers for the same intent. We call this phenomenon execution-correct structural divergence (ECSD). We introduce SQLStructEval, a framework that analyzes this behavior through canonical abstract syntax tree representations. We quantify structural diversity and agreement across generations. Experiments with different LLMs on multiple Text-to-SQL datasets, including Spider, document ECSD across models and datasets. Furthermore, our experiments demonstrate that generated queries are sensitive to question paraphrases and schema presentation. To address ECSD, we adopt a pipeline that first generates structured intermediate representations and then deterministically compiles them into SQL, improving execution accuracy and structural agreement among correct outputs. Structural analysis thus provides an additional diagnostic perspective that complements execution-based evaluation. Code is available at https://xanderzhou2022.github.io/AACL2026-SQLSTRUCTEVAL/.

Nüshu is an endangered phonetic script historically used by women in Jiangyong County, southern Hunan, China. While existing computational studies of Nüshu mainly focus on textual digitization and visual recognition, the acoustic reconstruction of its authentic pronunciation remains largely unexplored. Building a Nüshu text-to-speech (TTS) system is particularly challenging because available recordings are extremely limited and mostly consist of isolated syllable-level pronunciations rather than natural sentence-level utterances. In this work, we introduce NüshuVoice, the first TTS benchmark for Nüshu. We construct a sentence-level Nüshu text-to-audio dataset that aligns standardized Unicode Nüshu text, phonetic transcriptions, standard Chinese translations, and archival recordings. To synthesize speech under this extreme low-resource setting, we propose Nüshu-PitchVITS, an -conditioned VITS framework that leverages Nüshu’s five-level pitch notation as an explicit prosodic inductive bias. Experimental results show that Nüshu-PitchVITS outperforms strong TTS baselines in spectral fidelity, pitch reconstruction, and human-rated intelligibility. We publicly release the dataset and code at: https://anonymous.4open.science/r/Nvshu-TTS-2EB6.

Existing text-to-video retrieval benchmarks often permit matching from static appearance, making it difficult to determine whether models capture state changes and final outcomes. We introduce GenState-AI, a benchmark comprising 956 AI-generated videos and 195 verified state-change queries across Toy, Cartoon, Home, and Manipulation domains. Its construction uses prompt-controlled variants that target either conflicting temporal/state outcomes or semantic substitutions. Following human verification, all valid videos contribute to subset-level retrieval pools, while complete query-specific triplets support diagnostic hard-negative ordering. We evaluate VCF-Lik, Qwen3-VL, and LanguageBind using both standard Recall@K and triplet-based consistency metrics. Qwen3-VL (8B) achieves the best macro-average R@1 and triplet consistency, while LanguageBind obtains the highest semantic discrimination accuracy. However, stronger retrieval does not consistently yield correct hard-negative ordering, and Qwen3-VL and LanguageBind remain weak on Manipulation triplets, where VCF-Lik performs substantially better. GenState-AI therefore provides a controlled diagnostic testbed for distinguishing global retrieval quality from fine-grained temporal and semantic grounding in AI-generated videos. Dataset: https://huggingface.co/datasets/Mh-Li/genstate-ai

Large language models often generate plausible yet unreliable answers, making uncertainty quantification essential for detecting potential hallucinations. Prior work has shown that the information needed to detect hallucinations and quantify a model’s confidence in its generations may already be present in its internal states, but how to extract this information effectively remains an open problem. We introduce Semantic Coverage, a white-box, representation-level spectral uncertainty estimator inspired by coverage in reinforcement learning. Given multiple sampled generations for a prompt, Semantic Coverage quantifies uncertainty by measuring how broadly their internal representations span the model’s latent semantic space. Across three open-weights language models and four short-form question-answering benchmarks, Semantic Coverage achieves the strongest overall performance among the evaluated methods and consistently outperforms prior representation-level methods in quantifying LLM uncertainty.

Recent advances in Large Language Models have transformed Emotion Recognition in Conversation, yet existing instruction-tuned approaches still produce high-confidence misjudgments on utterances whose surface text is weakly informative but whose emotion depends on the surrounding situation. We propose SRERC, a selective reflective correction framework for LLM-based ERC. SRERC uses a verdict teacher to assess contextual support for the first-pass prediction and triggers reflective decoding only when reflection has an actionable role. For selected cases, an SRC generator constructs a Scene-aware Reflection Context, which provides target-specific situational grounding and verification probes for an SRC-conditioned second pass. This gated design enables targeted correction while preserving non-gated predictions and limiting unnecessary over-correction. Experiments on IEMOCAP, MELD, and EmoryNLP show that SRERC achieves the highest average weighted-F1 across the three benchmarks, with diagnostic gains on context-sensitive cases where surface cues are insufficient or misleading.

While neural machine translation (NMT) has achieved remarkable parity with human performance in general domains, handling multi-word expressions (MWEs) remains a persistent challenge, particularly in low-resource settings. This difficulty is twofold: models need not only decode the non-compositional meaning of MWEs but also navigate the stylistic choice between idiomatic ("MWE") and literal ("plain") outputs. In this study, we introduce a new 4-way parallel dataset for Greek and English, comprising of Greek and English MWE and plain sentences. We evaluate the style-transfer and style-switching capabilities of state-of-the-art Large Language Models (LLMs) across four tasks. Using traditional metrics, we demonstrate that LLMs often struggle with style-aware translation. Furthermore, we introduce StyleCOMET, a fine-tuned quality estimation metric to better evaluate stylistic nuances. Our results suggest that current limitations in MWE translation are a product of both model inability and metric-level insensitivity to idiomaticity.

Automated release note generation aims to reduce the cost of documenting frequent software updates. Existing benchmarks mainly rely on commit messages, limiting evaluation of how models use broader software-change evidence. We introduce ReleaseEval, a reproducible and openly licensed multi-granularity benchmark with 94,987 release notes from 3,369 repositories across six programming languages. Each release is aligned with commit messages, commit tree structures, and raw code diffs, supporting three task settings: commit2sum, tree2sum, and diff2sum. Automatic and human evaluations show that models benefit from lightweight commit-tree structure, while raw diffs remain challenging due to length and noise. These findings position ReleaseEval as a multi-granularity benchmark for studying release note generation. We release the dataset and code at https://github.com/Changanyuexia/rn_benchmark/

Recent large audio language models (LALMs) have achieved impressive progress in audio understanding. However, existing evaluations remain largely constrained to English and narrow audio domains. Prior benchmarks typically focus on a single audio modality, i.e., speech, sound, or music, limiting the systematic investigation into how these models generalize across diverse visual scenarios. In this paper, we introduce EXAM², a benchmark for multilingual and multimodal audio understanding spanning six languages and multiple modalities, including speech, sound, music, mixed-audio settings, and visual images. By incorporating visual information alongside heterogeneous audio inputs, EXAM² enables more realistic evaluation of scene-aware audio reasoning and cross-modal comprehension. EXAM² comprises 5,667 multiple-choice questions, 22,614 image instances, and 135,684 multilingual translations. We evaluate state-of-the-art open-source and proprietary LALMs as well as multimodal LLMs, revealing substantial performance gaps in multilingual and cross-modal understanding. Furthermore, we propose Gemma3n-EXAM², a lightweight fusion-model fine-tuned on EXAM²-train, achieves up to 15.8% improvement in multilingual settings and 16.5% gains in multimodal evaluation over a strong baseline. Empirical results establish EXAM² as a challenging benchmark and pioneer future multilingual and multimodal audio intelligence research.

Educational LLMs such as writing coaches, coding assistants, study companions, and AI tutors are widely marketed as student-centered, usually because they are “personalized” or because they raise performance on the assisted task. We argue that this is an unverified pedagogical claim: a system that helps students complete tasks is not necessarily a system that helps students learn. The distinction is between performance-centered assistance, which optimizes the immediate product, and student-centered learning, which preserves the learner’s cognitive work and validates capability the learner keeps without the system. A 2025 randomized trial shows the gap in practice: an unguarded tutor raised assisted performance while harming unassisted exam scores. We trace this to a units mismatch: “student-centered” is a learner-side claim, yet it is measured almost entirely with tutor-side proxies. We propose five auditable standards (S1–S5) that translate established learning-science principles into requirements for benchmark design, reward shaping, red-teaming, and deployment monitoring; a small illustrative process audit then shows both that the standards can be operationalized and why such audits cannot substitute for learner-side validation. Our position is that LLM-based educational systems should be judged not by answer quality, engagement, satisfaction, or assisted performance alone, but by whether they measurably support learners’ capacity to reason, self-regulate, transfer, and keep learning without the model.

We introduce the Image Reconstruction Game, a fully automated benchmark in which a vision-language model issues corrective instructions to an image generator across multiple turns, making accumulated common ground directly observable as a rendered image. Benchmarking two Describer models crossed with two Generator models across seven image categories, we find that the describer is the dominant factor in reconstruction quality, while the generator determines whether iterative refinement helps or hurts. Mathematical and geometric images pose the greatest challenge. The describer’s token budget strongly affects convergence: shorter budgets yield sparser first renderings with more room for visible improvement, while longer budgets raise absolute quality but leave less to fix. Stronger describers use a richer correction vocabulary spanning spatial, numeric, and structural categories, while weaker describers concentrate on surface properties and tend to stop after a few turns. Human validation shows that the best automated judge reaches only slight-to-fair agreement with human preferences, and automated scores require human recalibration to be used reliably.

Poetry generation in Sanskrit typically requires the verse to be semantically coherent and adhere to strict prosodic rules. In Sanskrit prosody, every line of a verse is typically a fixed length sequence of syllables adhering to prescribed binary patterns of syllable weights. We observe that instead of treating a verse as a monolithic sequence, segmenting them as grouped-lines leads to significant improvement in semantic coherence by 10% with comparable metrical adherence. Specifically, PINGALA, our proposed decoding approach is designed to encourage every line to have well-formed words and our token selection biases the model towards it by preferring longer tokens. Writing in Sanskrit follows phonemic orthography, hence using a phonetically aware transliteration scheme, SLP1, increased the metrical alignment by 46% with comparable semantic similarity, for a instruction fine-tuned large language models like Phi-4. We also introduce a new approach for reference-free evaluation using cross-encoders which achieved better alignment with true poetry instances.

Task-Oriented Dialogue (TOD) systems are commonly used to assist users in achieving specific goals through human-computer interactions. Existing methods typically employ a single model during response generation to simultaneously learn response style and perform fine-grained knowledge retrieval. However, these task-coupling methods often result in suboptimal responses generation and the inclusion of irrelevant knowledge in responses, making the optimization of TOD systems more difficult. To address this challenge, we propose a novel small-to-large model collaboration framework for task-oriented dialogue, named SMARTS. This framework utilizes a small language model to generate style-only responses (without knowledge), while a large language model retrieves top-1 relevant knowledge from the knowledge base independently. The style-only responses are finally filled with top-1 relevant knowledge and form the completed responses to users. Finally, a re-thinker is designed to check the language fluency and knowledge accuracy of responses. Experiments on two public datasets demonstrate that SMARTS outperforms existing methods by an average of 5.79% in Entity F1.

Low-bit activation-weight quantization is crucial for efficient large language model inference but remains fragile because activation quantization errors propagate across Transformer layers. We propose Activation Residual Hessian Quantization (ARHQ), a post-training residual-aware low-rank correction framework that directly targets this propagation. ARHQ measures the residuals produced by the deployed activation quantizer, builds an input-side residual Hessian, and identifies the weight directions that most amplify these residuals. This yields a closed-form truncated-SVD solution: residual-sensitive components are placed in a high-precision low-rank branch, while the remaining weights stay in the low-bit path. ARHQ requires no backpropagation, supports LoRA-style deployment, and complements activation smoothing. Across LLaMA-family models and Qwen3-4B-Thinking, ARHQ improves W4A4 reconstruction, reduces WikiText2 perplexity over matched-rank SVDQuant, preserves downstream reasoning and coding accuracy, and achieves up to 3.21× prefill speedup over BF16 (3.11× over AWQ W4A16) with over 60% memory saving.

Diagnosing the root causes of failures in Retrieval-Augmented Generation (RAG) is challenging, as current evaluations often conflate logical reasoning errors with the failure to adhere to retrieved context. To address this, we propose **ReFa**, a framework inspired by Structural Causal Models to explicitly disentangle *Reasoning* from *Faithfulness*. Through this lens, we characterize two distinct failure modes: the **‘Fool’**, representing a deficiency in logical reasoning, and the **‘Lazy’**, representing a lapse in faithfulness where the model defaults to parametric priors. To operationalize this diagnosis, we introduce Dual Reasoning Chain Editing, a mechanism that constructs controlled proxy chains to isolate reasoning structure from evidence faithfulness. We apply this method on **ReFaBench**, a novel benchmark constructed in this work featuring factual, counterfactual, and knowledge-conflicting scenarios. Furthermore, we propose **ReFa-DPO**, a decoupled preference optimization strategy that leverages these proxies to target specific failure patterns. Experimental results demonstrate that **ReFa-DPO** enhances robustness, particularly in mitigating parametric interference, by simultaneously enhancing both contextual faithfulness and reasoning capabilities.

Prompt rewrites can prescribe a change in an answer, not merely a change in wording. We introduce ORBIT-SIT, a framework that assigns each rewrite an explicit action on a task’s output space and separately measures correctness against transformed gold answers and consistency with transformed model predictions. Across eight operators, five datasets, and three instruction-tuned models, high task accuracy does not ensure compliance: Phi-4-mini achieves 90.7% accuracy on SST-2 but only 16.4% under a label-swap instruction. Deterministic validation and a three-annotator audit check the operator contracts. Non-deceptive binary controls retain a 17.8-percentage-point gap, and separating inference from transformation improves compliance, implicating joint execution rather than refusal alone. OrbitAlign, a quantized low-rank adaptation objective, reduces gold-grounded risk by 25.4 points on a 400-item BoolQ repair pool. The recipe lowers risk in 39 of 40 cells on two additional backbones, and same-action transfer across binary tasks reduces risk by 12.9–15.1 points. The binary tasks provide the primary evidence; structured tasks remain extensibility diagnostics. ORBIT-SIT makes declared prompt actions measurable and partly repairable without equating compliance with general understanding.

Latent thought pretraining augments training text with model-generated "thoughts". Because generating them is expensive, the obvious way to scale is to spend the budget only on "high quality" chunks — and the most intuitive quality signal is whether the thought generated before a chunk agrees with the one generated after. We show this intuition backfires. In a controlled 1.1B experiment that compares every allocation rule with a random baseline at its own inclusion rate, keeping only high-agreement chunks raises held-out perplexity above the budget-matched random baseline in every seed (+0.84%; n=5 vs. 3, exact permutation p=0.018). The deficit persists across a second training iteration, though downstream accuracy does not separate the filter from random at the same budget. We trace the cause: importance rises with agreement over most of its range, but the top agreement decile, where prior and posterior coincide or the prior is missing, holds degenerate latents that an agreement-only filter cannot exclude. To diagnose this we introduce PAIR, which pairs agreement with a length-corrected importance weight (raw importance is itself a length artifact, 𝜌=-0.786). Importance routing alone matches random at its budget, the intersection of the two filters is slightly worse than random (+0.3%, p=0.008), and inverting the importance signal collapses perplexity. Budget matching decides these verdicts: against a random baseline at a lower inclusion rate the agreement filter looks 4.9% worse and importance routing 1.2% worse, and an earlier version of this work credited the intersection with a gain that was likewise a budget effect. Intuitive single-axis quality filters do not beat random allocation, and any latent allocation study must compare each rule with a random baseline at its own inclusion rate.

Propaganda and persuasion technique detection, the task of identifying how misleading content persuades rather than merely what it claims, occupies a distinctive and underexplored position within computational misinformation research. This task requires interpretable and rhetorically grounded reasoning over persuasion techniques, making it particularly well aligned with neurosymbolic artificial intelligence (AI). This survey presents the first systematic review of neurosymbolic approaches for propaganda and persuasion technique detection, together with relevant structured-reasoning approaches from closely related areas that offer methodological foundations for these tasks. It synthesizes work spanning argument mining, knowledge-augmented learning, logic-based fallacy detection, structured reasoning with large language models (LLMs), and hybrid neural-symbolic architectures. We further organize existing methods through a unified taxonomy of neurosymbolic integration paradigms and review foundational rhetoric theory, benchmark datasets, and evaluation methodologies. We identify key open challenges, including subtle and implicit persuasion, cross-lingual generalization, explanation faithfulness, and the growing challenge of LLM-generated propaganda. Finally, we outline future research directions and argue that neurosymbolic AI provides a promising foundation for bridging predictive performance with human-interpretable rhetorical reasoning in next-generation misinformation detection systems.

Evaluating text-to-image (T2I) systems requires judging not only whether an image matches a prompt, but also whether socially salient attributes are represented faithfully and without unsupported inference. Existing automated evaluators typically rely on face-centric recognizers or contrastive image–text similarity, which provide limited diagnostic feedback and often force predictions even when visual evidence is ambiguous or absent. For fairness-sensitive attributes such as religion and disability, where cues may be contextual, indirect, or intentionally unspecified, these evaluators can therefore miss failure modes that careful human reviewers would notice.

We introduce FairJudge, an abstention-aware evaluation protocol that uses instruction-following multimodal LLMs as structured judges for social-attribute prediction, profession grounding, and prompt–image alignment. The protocol constrains outputs to closed label sets, requires visible-evidence rationales, supports an explicit unspecified decision when cues are insufficient, and maps rubric-based alignment judgments to . These constraints turn MLLM judging from open-ended assessment into a parseable, auditable evaluation procedure.

Across four attribute-prediction benchmarks and three profession/alignment benchmarks, FairJudge outperforms or complements CLIP, DeepFace, VIEScore, and VQAScore. Ablations show that closed labels, abstention, and evidence reporting are central to reliability. We further introduce DIVERSIFY and DIVERSIFY-Professions, two context-rich resources for evaluating social representation and profession grounding beyond face-visible or iconic cues. We release code, prompts, datasets, parser logs, and per-image judge outputs to support reproducible auditing.

LLM agents that accumulate procedural memory can reuse successful plans, but what happens when the world changes? Deprecated APIs, revised clinical thresholds, and reorganized knowledge graphs silently invalidate stored procedures. No existing benchmark evaluates whether a memory system can maintain its procedures under such drift. We introduce WK-Drift, a benchmark constructed from three public temporal datasets (Time-Sensitive QA, GitChameleon, DriftMed). WK-Drift formalizes drift as typed events with a fact-change/procedure-change taxonomy, defines a four-phase protocol (learn, drift, evaluate, recover), and introduces entity-plan linkage for targeted maintenance. We evaluate ten configurations spanning no-memory, unstructured (including Mem0 and MemRL), tagged, and graph-based approaches. Graph-based surgical updates achieve 90% post-drift accuracy on knowledge QA (+6pp over stale baselines), 64% on clinical (+26.5pp), and 57% on code (+7.4pp), reducing staleness from 77–86% to 41–42%. The advantage concentrates on procedure-change tasks (+17.4pp) and scales with propagation depth (+50pp at depth 3). Surgical maintenance achieves 59% from the first post-drift epoch versus 26% for stale baselines, eliminating multi-epoch recovery latency.

Evaluating the faithfulness of multi-sentence summaries generated by language models (LMs) remains challenging due to their length and the subtlety of factual errors. While claim decomposition methods have proven useful for assessing short summaries (1-3 sentences), their effectiveness for longer, coherent summaries across diverse domains is not well understood. To better understand this issue, we first conduct an empirical study analyzing the quality of claims produced by existing decomposition approaches. Motivated by our findings, we propose a new discourse-informed method, Describe, which first extracts claims from the full summary using discourse structure, and then evaluates them through a structure-guided process that progresses from intra- to inter-sentential claim validation, enabling more fine-grained faithfulness evaluation. Experimental results across benchmarks reveal the benefits of discourse-guided decomposition and evaluation. Further analysis of model-generated error rationales and ablation studies shows our method effectively identifies nuanced errors within sentences, particularly those arising from complex discourse structures.

In-context learning (ICL) allows large language models (LLMs) to adapt to new tasks from a few examples, making it promising for languages underrepresented in pre-training. Recent work on many-shot ICL suggests that modern LLMs can further benefit from more ICL examples enabled by their long context windows. However, the inference cost can be prohibitive for low-resource language communities, and it is unclear whether example selection can reduce the number of examples needed. In this paper, we present a systematic empirical study of many-shot ICL for machine translation from English into ten truly low-resource languages recently added to FLORES+. We analyze the effects of retrieving more informative examples, using out-of-domain data, and ordering examples. Our findings show that many-shot ICL becomes more effective as the number of examples increases. More importantly, we show that BM25 retrieval substantially improves data efficiency: 50 retrieved examples roughly match 250 many-shot examples, while 250 retrieved examples approach the performance of 1,000 many-shot examples. We further show that example ordering has little effect and that ICL provides additional gains on top of fine-tuning.

Large language models (LLMs) struggle with relation completion (RC), both with and without retrieval-augmented generation (RAG), particularly when the required information is rare or sparsely represented. To address this, we propose a novel multi-stage paraphrase-guided relation-completion RAG framework, RC-RAG, that systematically incorporates relation paraphrases across multiple stages. In particular, RC-RAG: (a) integrates paraphrases into retrieval to expand lexical coverage of the relation, (b) uses paraphrases to generate relation-aware summaries, and (c) leverages paraphrases during generation to guide reasoning for relation completion. Experiments with five LLMs on two benchmark datasets show that RC-RAG consistently outperforms several RAG baselines. In long-tail settings, the best-performing LLM augmented with RC-RAG improves by 50.4 Exact Match (EM) points over its standalone performance and surpasses two strong RAG baselines by 30.4 and 16.0 EM points, respectively, without requiring additional model training or fine-tuning.

Causal discovery from unstructured data is a challenging yet underexplored task in high-expertise domains such as healthcare, finance, and education. Existing methods typically leverage the general knowledge of large language models (LLMs) to identify causal factors from unstructured data and annotate them into structured data for causal graph construction. However, they remain limited by two key challenges: (CH1) insufficient identification of latent factors, which are implicit in the data yet essential for causal discovery, and (CH2) unreliable factor annotation, causing errors to propagate into the resulting graphs. To address these challenges, we introduce a novel Domain Knowledge-enhanced Causal Discovery framework (DKCD) for extracting causal graphs from unstructured data in high-expertise domains with three interconnected components: (1) Knowledge Mining: It retrieves relevant domain knowledge based on observable factors to support causal reasoning. (2) Knowledge-guided Causal Reasoning: Reasoning with relevant knowledge discovers latent causal factors to address CH1 and generates causal clues for more accurate annotation to address CH2, and (3) Causal Structure Discovery: It constructs the final causal graphs based on a more complete factor set and accurate annotations. Experiments on two domain-specific datasets show that DKCD significantly improves both causal factor identification and causal graph construction. The code and data are available at https://github.com/bigoldli329/DKCD.

Reinforcement learning has become a common approach for improving language-model reasoning, but its effectiveness depends heavily on reward design, especially for compact domain-specialized models. We study structured biomedical question answering with two 4B-scale models, MedGemma-1.5-4B-it and Qwen3-4B-Thinking-2507, using a two-stage SFT+GRPO pipeline that trains models to produce tagged think/answer outputs. We compare three reward schemes: a simple format + accuracy reward, a depth incentivized reward that encourages multiple reasoning cycles, and a gated LLM-judge reward for intermediate reasoning milestones. Across five biomedical QA benchmarks, the simplest reward is the most robust: it substantially improves MedGemma in the structured-output setting, while depth and LLM-judge rewards often reduce accuracy. For Qwen3, SFT or native long-budget inference can outperform GRPO on several benchmarks, showing that reasoning-tuned models may not benefit from additional structural reinforcement. Our results suggest that, for lightweight biomedical QA, reward simplicity and model priors are more important than increasingly elaborate intermediate reasoning supervision. The codebase for this study is publicly available on GitHub: https://github.com/batuinmetz/When-more-supervision-hurts.

Prior work on climate misinformation detection typically classifies individual comments or claims using text alone. This framing overlooks a key property of online climate discourse: misinformation is often produced, reinforced, and sustained through recurring rhetorical patterns that reappear across comments and threads. Using a large-scale TikTok climate discourse dataset, we construct discourse profiles by clustering climate-related comments into recurring rhetorical patterns. We then incorporate these profiles into fine-grained climate misinformation classifiers using a late-fusion architecture. Under video-disjoint evaluation across three training seeds, discourse-profile-aware models significantly improve macro-F1 over text-only baselines for bag-of-words, TF-IDF, and MPNet representations. RoBERTa-large exhibits a positive mean improvement but substantially greater variability, and the difference is not statistically significant. Beyond classification, our results show discourse profiles shape misinformation reception and propagation. Discourse-profile-conditioned language model agents exhibit misinformation acceptance patterns aligned with profile-level misinformation prevalence in ClimateDisc, and the relationship between misinformation prevalence and engagement persistence differs across dominant discourse profiles.

Culture shapes how people interpret and navigate social interactions, yet existing Indonesian cultural commonsense benchmarks evaluate LLMs on short and isolated prompts, stripping away the dialogic context in which cultural nuances actually surface. We introduce CULTURETALK-ID, the first dialogue-based benchmark for cultural commonsense in Indonesian and its local languages, comprising 4,496 culturally grounded dialogues across 11 languages and 13 culturally salient topics, curated through a multi-stage human pipeline with native speakers to ensure authenticity. CULTURETALK-ID introduces three complementary tasks, namely dialogue-based multiple choice cultural commonsense reasoning, culturally faithful machine translation, and language steering, which jointly probe whether LLMs can understand, transfer, and generate culturally grounded language. Our results show that open-source models still underperform, particularly in the generation task and local language settings, highlighting a gap in their ability to capture and generate culturally grounded knowledge in conversational contexts.

Automated research systems support literature synthesis, ideation, experiments, writing, and peer review, but their evaluation is dispersed across tasks, benchmarks, and studies that are difficult to compare directly. We review this literature from the perspective of evaluation design and evidence, covering six targets: literature synthesis, research ideation, executable workflows, scholarly writing and communication, automatic peer review, and end-to-end research. We compare task construction, evidence sources, evaluators, and scoring procedures to explain the capabilities assessed by different designs. Our synthesis highlights three recurring lessons: output checks, process checks, and human studies provide complementary information; evaluator calibration is specific to the property being assessed; and resource budgets and attempt selection are integral to interpreting performance comparisons. We identify diagnostic evaluation designs and documented gaps in supporting evidence, and translate these comparisons into reporting and audit recommendations for specific evaluation settings. The survey helps readers navigate existing evaluations, select appropriate benchmarks, and design subsequent studies.

Transformer language models predict the next token by passing through dozens of sequential layers, yet whether adjacent layers operate independently or coordinate their computations—and how this relationship forms during pretraining—remains largely unknown. We view residual updates of adjacent layers as vector fields and introduce a Lie bracket-based inter-layer coordination metric that quantifies how much two layers depend on each other’s outputs. Analyzing pretraining checkpoints of 6 models across 3 architecture families (Pythia, OLMo, Amber), we find that all layer pairs are nearly independent at initialization, but as pretraining progresses, individual layer contributions decrease while coordination systematically increases—concentrating in the middle layers that existing metrics (BI, CKA) classify as “redundant.” The mechanism is Jacobian amplification: middle layers write small updates but respond sensitively to their neighbors, and this amplification co-emerges with coordination (r > 0.99). Block duplication experiments confirm that high-coordination blocks are 2.7–7.9 more robust than low-coordination blocks, verifying that this cooperative structure substantively contributes to model function. Our findings reframe pretraining as a process that builds cooperative circuits between layers rather than strengthening each layer individually.

LLM-augmented symbolic planning combines natural-language goal grounding with symbolic search and constrained decoding. We present Abstraction-based Grammar-Constrained Planning (AGCP), a framework that induces lifted action schemas from environment rollouts, compiles a bounded language of grounded action sequences that are applicable and goal-reaching under the induced model, and enforces that language during decoding. The guarantee is relative to the induced model. Environment execution additionally requires a sound induced model and a correct parsed goal. With the model, prompts, instances, and decoding settings fixed, replacing a syntax-only grammar with the applicability mask, compiled here from the domain’s own transition model, moves goal-reaching samples from 0/160 to 160/160. On human-annotated ALFWorld goals, verifier-guided correction raises parse acceptance from 61.4% to 75.4% and execution success from 67.5% to 80.7% on 114 paired tasks. Under paraphrasing, GPT-4o and Sonnet grounding remain stable while a regular-expression baseline falls from 31.9% to 0.8%. Together, these results locate validity relative to the induced model in an applicability grammar built from a symbolically observable, bounded environment. In this setting, natural-language goal grounding is the main remaining role of the LLM.

Large language models (LLMs) inevitably exhibit hallucinations since the accuracy of generated texts cannot be secured solely by the parametric knowledge they encapsulate. Although retrieval-augmented generation (RAG) is a practicable complement to LLMs, it relies heavily on the relevance of retrieved documents, raising concerns about how the model behaves if retrieval goes wrong. To this end, we propose Corrective Retrieval Augmented Generation (CRAG), an autonomous agent that dynamically corrects and refines the retrieval content. Specifically, an agent core is introduced that perceives, decides, and executes in a closed loop. It dynamically selects and orchestrates tools, including internal retrieval, web search, keyword evaluators, semantic evaluators, and LLM evaluators, to perform multi-dimensional contextual quality evaluation and knowledge supplementation, thereby constructing accurate and precise retrieval content. Furthermore, because CRAG refines and corrects retrieved context without requiring modifications to the underlying system, it naturally serves as a plug-and-play framework that can be seamlessly integrated with a wide range of RAG-based approaches. Experiments on five datasets demonstrate that CRAG outperforms previous approaches, while its autonomous decision-making capabilities improve tool-calling efficiency and retrieval context quality.

We present the first cross-platform computational analysis of AI-sustainability discourse across LinkedIn, X, and Bluesky, using 52,102 posts collected over 15 months. Combining RoBERTa-based sentiment classification with BERTopic topic modeling, we identify 36 topics and compare sentiment, topic distributions, and engagement across platforms. Overall, 43.98% of posts are positive, 40.16% neutral, and 15.86% negative. Sentiment varies by topic rather than by platform: corporate and AI-potential themes are mostly positive, while resource-cost topics carry the highest negative sentiment. Topic distributions differ significantly across platforms: LinkedIn focuses on corporate sustainability, X on hardware and cryptocurrency, and Bluesky on environmental resource costs. Niche topics consistently attract more engagement than high-volume ones. These findings have implications for how organizations and policymakers communicate about AI-sustainability, as topic framing and platform audience composition shape public discourse more than platform selection alone. We release the annotated dataset to support future research.

Large-scale ASR models such as Whisper perform well in general domains but struggle with specialized terminology, particularly in sensitive domains like medicine. We propose WhisperX-LLM-TermFixer, a lightweight post-decoding framework that improves term recognition without retraining or modifying the acoustic model. The method combines N-best decoding, phonetic n-gram correction, and LLM-based contextual filtering. We evaluate the method on a controlled synthetic benchmark containing 1,005 utterances per language in English, Spanish, Catalan, and Basque, with one target medical term per utterance and full lexicon coverage. Across six Whisper sizes, the selected configurations improve term accuracy by 16.12-35.32 percentage points and reduce WER by 0.88-4.01 points relative to WhisperX. LLM filtering reduces WER relative to the phonetic stage in all evaluated configurations, although it reduces term accuracy in three Basque configurations. Perplexity results provide additional evidence of improved linguistic plausibility. The selected configurations require approximately 1.26x-2.38x the baseline inference time. Our code and data are publicly available at https://github.com/NLP-UMUTeam/WhisperX-LLM-TermFixer.

How much of large language model (LLM) performance in code analysis reflects memorization of training data rather than genuine reasoning? We introduce BLOCKBENCH, a contamination-controlled evaluation framework that separates memorization effects from analytical capability in smart contract vulnerability detection. We evaluate seven frontier models on 316 vulnerable contracts augmented with 322 semantic-preserving transformations and 100 clean contracts. We observe a substantial performance gap across data exposure levels: detection accuracy drops from 86.5% on likely-contaminated samples to 27.6% on post-cutoff contracts, and further to 0.8% on the most recently disclosed subset. A temporal analysis shows a strong exposure effect, where detection rates increase from 5.9% to 84.7% as contracts become more publicly visible, indicating that benchmark performance is heavily inflated by data contamination.

We further identify a pattern matching paradox, where models can often locate security-relevant code regions but fail to explain why they are exploitable, revealing a form of explanation unfaithfulness in code understanding. In contrast, traditional static analyzers show complementary limitations: Slither compiles only 6.6% of post-cutoff contracts, while Mythril compiles all inputs but fails to detect vulnerabilities. BlockBench generalizes to code understanding tasks where semantic equivalence can be verified, enabling more reliable evaluation under potential contamination.

Retrieval-augmented generation (RAG) reuses the same documents across many requests, making non-prefix KV-cache reuse attractive. Yet a cached document state is not reusable text: it is a context-conditioned causal state whose keys and values depend on the prefix, position, and preceding documents under which it was computed. We formulate non-prefix KV reuse as a behavior-preservation problem with full prefill on the exact prompt as the reference, and introduce SAFE-KV, which repairs a composed cache by re-encoding the causal suffix from a selected start document. On Qwen2.5-7B-Instruct, prefix-conditioned composition preserves only 47.2% of HotpotQA-500 and 37.8% of 2Wiki-500 full-prefill answers; SAFE-KV with risk-guided K2 selection restores 99.8%/98.4% (donor-copy protocol) and 100.0%/98.4% with actual suffix forwarding. Document-only and fixed-window repairs recover far less, because the document states after the repaired region keep histories that omit it. Reuse without repair is exact up to four documents and degrades at sixteen, where suffix repair recovers 98–100%. Faithful repair re-encodes most of the context, so in a single-request L40S study SAFE-KV’s operating rule serves every workload by full prefill. SAFE-KV thus establishes the causal-state contract for non-prefix reuse and a validated repair mechanism that meets it.

Legal documents are often lengthy and contain domain-specific terminology and complex legal reasoning, making them difficult for the general public to understand. This challenge is further amplified in the Indian context, where a large portion of the population lacks proficiency in complex English. Although automatic legal summarization has been studied, most existing approaches generate summaries intended for legal professionals and thus remain difficult for laypeople. To address this gap, we introduce InLegalLaySum, the first benchmark for lay summarization of Indian legal documents. InLegalLaySum comprises 9,570 judgements from the Supreme Court and several High Courts of India paired with expert-written summaries in English. We benchmark a wide range of generative models (ranging from 160M to 7B parameters) and perform comprehensive experiments with multiple training strategies, including supervised fine-tuning (SFT) and direct preference optimization (DPO). We observed that the hybrid SFT+DPO training achieved superior performance across readability metrics, which is further validated by domain experts.

Large language model (LLM)-based agents are widely deployed in user-facing services but remain error-prone in new tasks, tend to repeat the same failure patterns, and show substantial run-to-run variability. Fixing failures via environment-specific training or manual patching is costly and hard to scale. To enable self-evolving agents in user-facing service environments, we propose WISE-Flow, a workflow-centric framework that converts historical service interactions into reusable procedural experience by inducing workflows with prerequisite-augmented action blocks. At deployment, WISE-Flow aligns the agent’s execution trajectory to retrieved workflows and performs prerequisite-aware feasibility reasoning to achieve state-grounded next actions. Experiments on ToolSandbox and 𝜏²-bench show consistent improvement across base models.

While retrieval-augmented generation (RAG) grounds large language models (LLMs) in external knowledge, ensuring faithful evidence utilization in complex domains like law remains challenging. LLMs often ignore retrieved facts or indiscriminately copy from noisy contexts. Existing context-aware decoding methods attempt to mitigate this by amplifying the influence of retrieved content, but they treat the context as a monolithic block—amplifying irrelevant spans alongside evidence—and intervene at every decoding step, incurring substantial inference overhead. To address this, we propose SaCoD (Selective and Confidence-aware Decoding), a training-free inference framework. SaCoD leverages the LLM’s internal attention signals to dynamically isolate salient evidence paragraphs at each step, restricting a customized copy mechanism strictly to relevant facts. Furthermore, an entropy-based gate activates this intervention only when the model is uncertain, bypassing copy operations for deterministic tokens to preserve linguistic fluency and cut computational overhead. Experiments on four legal question-answering benchmarks across three open-weight LLMs demonstrate that SaCoD significantly improves faithfulness and response correctness while substantially reducing inference latency compared with prior context-aware methods.

Uncertainty quantification (UQ) methods are widely used for hallucination detection in large language models (LLMs) in closed-book settings where ground-truth evidence is unavailable at inference time. Prior work has proposed combining UQ signals via learned ensembles, but empirical investigations into the robustness of these ensembles are limited. We study a supervised ensembling framework that trains a classifier over heterogeneous UQ-based scorer outputs on a small, domain-specific dataset of labeled LLM responses, then applies it to out-of-sample hallucination classification without retrieval, tools, or reference documents. Across four LLMs, nine datasets, and three generation regimes (short-form QA, long-form generation, and code generation), we provide a systematic robustness analysis along three axes: sample efficiency, in-domain dataset transfer, and generation regime dependence. We find that supervised ensembles outperform the best individual scorer in 30 of 32 settings, with gains realized from as few as 100 labeled instances. Ensembles retain most of their advantage in cases of in-domain transfer under distribution shift, outperforming the best non-ensemble scorer in 23 of 28 transfer settings. Sampling-based black-box ensembles are nearly as effective as full ensembles, while single-generation white-box ensembles offer limited benefit.

Personalization of LLMs can greatly improve user experience by catering to individual preferences and contexts. However, unnecessary differences in model behavior across users based on their personal information can be undesirable. Such differences introduce personalization bias, leading to inconsistent or unsafe behavior across social attributes such as religion, race, and gender. We introduce (Fairness Aware Imbalance Reduction), a framework for mitigating personalization bias by addressing imbalances in stereotype-related content during preference optimization. includes two variants: _D, which applies data-level intervention based on data augmentation with LLMs, and _T, which incorporates fairness-aware reweighting directly into the optimization objective. We evaluate both variants using multiple personalization bias metrics across diverse identities and models. Across all three LLMs, both methods consistently reduce bias relative to baseline, with up to 20 and 30.1 relative improvement by _D and _T, respectively.

Vision-language models (VLMs) frequently fail at visual change reasoning, even when their vision encoders contain sufficient information. We observe that correct VLM outputs tend to contain explicit verbal evidence (object names, colors, spatial locations) that supports the claimed change, while incorrect outputs often lack such evidence. We propose SAVER (Selective Auditing of Verbal Evidence for Error Recovery), a lightweight, rule-based method that parses VLM responses for this evidence and triggers structured reprompting only when evidence is missing or inconsistent. Across three change detection benchmarks and four VLMs, SAVER significantly improves accuracy on tasks where errors stem from the model failing to articulate what it saw (expression failures), with gains up to +25.8% on CLEVR-Change. The evidence patterns can also be generated by an LLM in a single call, matching the hand-tuned gate on CLEVR-Change. Ablation experiments confirm that the evidence gate, not reprompting alone, drives the improvement.

Multi-Modal In-Context Learning (MM-ICL) is commonly employed to enhance performance of Large Vision-Language Models (LVLMs) on downstream tasks. However, as next-generation models like QwenVL and InternVL have been optimized through instruction tuning and alignment, their baseline capabilities have been strengthened. This necessitates a re-evaluation of the effectiveness and optimal configuration of MM-ICL across different models. By evaluating diverse LVLMs spanning different parameter scales (7B–32B) and accessibility settings (open-source and closed-source) across eight cross-domain datasets, this paper systematically investigates the performance of MM-ICL under varying model capabilities and task settings. Experimental results indicate that, beyond validating established patterns, next-generation models exhibit distinct phenomena: in certain logical reasoning and format-sensitive tasks, the introduction of MM-ICL does not necessarily yield improvements and can even lead to performance degradation in specific scenarios. We analyze these observations and, based on extensive experimental data, provide new recommendations for selecting MM-ICL strategies for LVLMs.

Multilingual language models can answer factual questions across languages, but their answers are not always consistent across languages, and the internal pathways used to retrieve this knowledge remain unclear. A model may rely on language-specific shortcuts, shared multilingual representations, or another language as an internal pivot. We investigate these possibilities in Llama-3.1-8B and Gemma-2-9B with sparse autoencoders (SAEs), identifying, analyzing, and causally intervening on language-associated features during factual recall. On CUS-QA (CITATION), a parallel QA dataset spanning Czech, Slovak, Ukrainian, and English, we identify SAE features associated with particular subsets of languages. We then decompose each question’s activations into these subsets to measure reliance on input-language, multilingual, or pivot-like representations across layers. Across both models, factual recall is heterogeneous: different input languages and questions exhibit different language-associated pathways. Steering substantially changes cross-lingual answer alignment and sometimes increases correctness in Llama, while it produces only small changes in Gemma. These results show that multilingual factual recall is mediated by identifiable SAE features that can be used to diagnose and control how models access knowledge across languages.

Existing hallucination taxonomies classify LLM errors by what is wrong with the output — memorised misconceptions, reasoning failures, fluent fabrications — but cannot answer a different question: which uncertainty scorer would have caught this error? We propose a complementary taxonomy that classifies errors by their detectability signature, the signal a scorer family would read. The DECK taxonomy is a 2×2 partition along inter-sample consistency and token-level confidence into four regimes (Drift, Entrenched, Confabulation, Knotted) that yields a falsifiable blind-spot map: black-box consistency scorers have signal in D and C, white-box token-probability scorers in K and C, and only an LLM-as-a-Judge with independent pretraining can detect E. Across three models and four short-form QA datasets we test this map two ways: judge-involving scorer disagreements concentrate in each family’s predicted blind-spot cells, and external labels (SelfAware unanswerable, HaluEval adversarial, PopQA entity popularity) land in the predicted cells, robustly to cross-fitted thresholds. We further identify a universal blind spot of output-level UQ: on knowledge-gap inputs where the generator emits confident, repeatable fabrications, every output-level family collapses by construction. A linear probe on Llama-3-8B’s final-layer hidden states also falls to chance, with or without quantisation, though an intermediate layer retains weak signal.

Recent Large Audio Language Models (LALMs) demonstrate impressive performance on speech benchmarks, yet it remains unclear whether these models genuinely process acoustic signals or rely on text-based semantic inference. To systematically study this question, we introduce DEAF (Diagnostic Evaluation of Acoustic Faithfulness), a bench- mark of over 2,700 conflict stimuli spanning three acoustic dimensions: emotional prosody, background sounds, and speaker identity. Then, we design a controlled multi-level evaluation framework that progressively increases textual influence, ranging from semantic conflicts in the content to misleading prompts and their combination, allowing us to disentangle content-driven bias from prompt-induced sycophancy. We further introduce diagnostic metrics to quantify model reliance on textual cues over acoustic signals. Our evaluation of seven LALMs reveals a consistent pattern of text dominance: models are sensitive to acoustic variations, yet predictions are predominantly driven by textual inputs, revealing a gap between high performance on standard speech benchmarks and genuine acoustic understanding. Code and benchmark resources are publicly available at: https://github.com/kikixiong/DEAF

Evaluating LLMs as math tutors is challenging because many automated protocols either score a single response against a reference or place the model in a fixed dialogue context, without measuring its ability to build and steer tutoring interactions. We present a fully automated, simulation-based framework that generates controlled multi-turn teacher–student dialogues with turn-level answer verification under a fixed interaction budget. This budget specifies the maximum number of student–teacher rounds allowed per problem (K=5 in our main experiments). To prevent score inflation from answer-revealing feedback, we introduce an overinformative eligibility filter: a hint-only student test applied to the teacher utterance immediately preceding the student’s first correct attempt. Only eligible corrections receive credit in our effectiveness and efficiency metrics, both of which are normalized by the number of initially incorrect sessions. We also propose a diagnostic metric suite separating tutoring effectiveness/efficiency (ECR/FEI), answer reachability (ARG), and teacher judgment (correctness/incorrectness F1). Experiments on GSM8K and MATH with diverse teacher and student models show that problem-solving accuracy is an informative but incomplete proxy for interactive tutoring ability, and that filtering overinformative feedback can substantially alter apparent performance. Our framework enables scalable, reproducible, and fine-grained evaluation of LLM math tutoring beyond outcome-only scoring.

As LLMs increasingly assist in moral reasoning, omission bias, the tendency to prefer inaction even when equivalent framings reverse substantive outcomes, poses a significant risk of skewed decision-making. Yet omission bias remains underexplored in LLM evaluation, with the few existing studies limited in scale and focused largely on utilitarian-deontological conflicts. To address this gap, we introduce OMIT, a benchmark consisting of 218 paired-frame scenarios across 10 conflict types, constructed by leveraging disagreement patterns from an LLM-based, five-perspective philosophical persona panel (utilitarianism, deontology, virtue ethics, care ethics, and contractualism). Evaluating eight LLMs, we find that omission bias is pervasive but inversely correlates with model size within families. We further evaluate four inference-time interventions and find that interventions encouraging models to consider moral principles before committing to a yes/no answer reduce omission bias and increase frame-consistent responses, although lower omission bias rates can also coincide with shifts toward action-biased responses. Ultimately, this work contributes not only the OMIT benchmark, but also a methodology for using diverse philosophical disagreement signals to evaluate framing-sensitive inaction preferences and the distributional effects of mitigation attempts in LLMs under complex moral conflicts.

Reliable tool use requires more than triggering a mechanism or matching a query to an API description. Before selecting a specific tool, an agent must first infer the capability requirements implied by the user query. In this paper, we investigate whether these query-side capability requirements are linearly decodable from LLM hidden representations prior to generation, and how this hidden-state accessibility compares with explicit verbal classification. We introduce TACIT, a framework that decomposes external requirements along three fundamental axes: Source, Transformation, and World Effect, defining eight structurally distinct capability classes. Using 1,600 balanced training queries from benchmarks, synthetic examples, and new domain scenarios, we train linear probes on pre-generation hidden states from four open-weight LLM families. Our empirical results demonstrate that fine-grained capability structures are linearly decodable with high accuracy across all models. Crucially, however, we expose a representation-to-verbalization gap: these same models are significantly less reliable when asked to explicitly classify the same queries in natural language. This disconnect indicates that information about required external capabilities is linearly accessible in LLM hidden representations but not reliably expressed—a phenomenon we define as structured but silent.

Cognitive distortions are unhelpful thinking patterns central to Cognitive Behavioral Therapy (CBT) and closely linked to mental health outcomes. Despite their context-dependent nature, studies at the intersection of Natural Language Processing and Cognitive Behavioral Therapy are centered largely around sentence-level utterances, ignoring the document-level complexity of real clinical text. To this end, we propose a two-stage rank-then-classify cascade that decomposes document-level multi-label cognitive distortion detection into (1) a sentence-ranking stage that identifies the most likely distorted sentences, and (2) a multi-label classification stage that assigns distortion subtypes to the selected sentences by incorporating their surrounding interleaved context. While the pipeline stages are trained on separate out-of-domain datasets, our model matches state-of-the-art performance on the CBT-CD benchmark of 166 expert-annotated clinical vignettes. Operating with only 368M parameters, our system achieves a 51.8% weighted F1 (9-class), surpassing massive large language model baselines such as Mistral-7B-v0.3 (46.0%) and performing on par with state-of-the-art models such as Gemma 4 26B A4B (50.8%).

Large language models (LLMs) are increasingly deployed for enterprise information extraction (IE), where the same document must be reorganized differently for each user. Existing prompt optimization methods, however, rely on a single prompt optimized against a global objective, which is misaligned with the inherent user heterogeneity of real workplaces. We formulate enterprise IE as per-user prompt adaptation under interaction feedback and propose Self-Meta-Evolve, a hierarchical framework that maintains a dedicated prompt for each user and continuously refines it through a dual-loop process: an inner loop that edits structured prompts based on persona-conditioned feedback, and an outer loop that evolves the meta-prompt itself by distilling successful editing patterns. To enable scalable training and evaluation, we release a persona-driven IE benchmark of 292 simulated enterprise users, paired with a reproducible persona-generation pipeline grounded in O*NET occupational taxonomies. On this benchmark, Self-Meta-Evolve achieves a 74.58% success rate, outperforming the strongest prompt-optimization baseline by 13.56 absolute points, and reaches 52.54% within only two iterations. A double-blind human study with twenty real professionals further confirms that prompts adapted by our framework win against static baselines in 71% of pairwise comparisons.

Pretrained text encoders are prone to shortcut learning, relying on token-label correlations that fail once the distribution shifts in deployment. Existing shortcut mitigation methods mainly operate at training time and assume access to training data, training dynamics, or shortcut annotations, which are hardly available during deployment, where only the converged model remains. We show that this model alone suffices to mitigate shortcuts during deployment: a biased model internalizes a signal of its learned shortcuts that can be captured via unsupervised gradient-based attribution. We further prove that deployment-time mitigation is information-theoretically upper-bounded by training-time mitigation. Nevertheless, exploiting this gradient signal, our proposed unsupervised deployment-time shortcut mitigation framework for pretrained text encoders, Shortcut Guardrail, recovers substantial performance under shortcut distribution shift, matching or outperforming training-time baselines across sentiment classification, toxicity detection, and natural language inference.

Automated fact-checking has been a challenging task for the research community. Prior work has explored various strategies, such as end-to-end training, retrieval-augmented generation, and prompt engineering, to build robust fact-checking systems. However, their accuracy has not been high enough for real-world deployment. We, on the other hand, propose a new learning paradigm, where evidence classification and entailed justifications made by generative language models (GLMs) are used to train encoder-only language models (ELMs). We conducted a rigorous set of experiments, comparing our approach with recent works along with various prompting and fine-tuning strategies. Additionally, we performed ablation studies, error analysis, quality analysis of model explanations, and a domain generalisation study to provide a comprehensive understanding of our approach.

Character-level perturbations are widely used to evade Chinese toxicity detection, yet existing benchmarks either cover only a limited range of perturbations or expand perturbation coverage without balanced construction across toxic and non-toxic samples, leaving these two requirements unmet within a single benchmark. We introduce COAT (Chinese Orthogonal Adversarial Toxicity), a 41K-sample benchmark spanning eight perturbation types that enforces perturbation orthogonality: all perturbations are applied with equal probability to both toxic and non-toxic texts, ensuring P(toxic | perturbed) = 0.5. As a core diagnostic layer, we annotate Natural Language Variant Ambiguity (NLVA): 21.6% of clean samples naturally contain surface features overlapping with perturbation artifacts, enabling controlled analysis of model sensitivity to perturbation-like features and classification difficulty. Evaluating five encoder classifiers, three proprietary LLMs, and Llama3-8B with chain-of-thought reasoning, we find that (1) models remain systematically sensitive to perturbation-like features even under balanced training, and (2) across Llama3-8B variants, overcorrection varies systematically with the linguistic ambiguity introduced by each perturbation type. Step-level diagnosis localizes the shared bottleneck to semantic restoration, with targeted preference optimization yielding reliable gains on the most ambiguous types.

Decoding-time mitigation of vision-language model (VLM) hallucination applies one fixed intervention and typically trades recall for lower hallucination. We trace this trade-off to treating hallucination as a single failure mode. Measuring the normalised entropy H of leave-one-out attributions over panoptic image segments, we find that a VLM’s inference mass concentrates in one task-conditional tail of H: captioning in the over-concentration tail (low H, 53–66%), binary question-answering in the over-spread tail (high H, 74–95%). The pattern holds in 6/6 (backbone × benchmark) cells on LLaVA-1.5-7B and Qwen2.5-VL-7B, survives exact Shapley recomputation, reproduces within AMBER, and is shared by three further backbones. Because errors and correct outputs share the tail—H is a regime signal, not a hallucination detector—an intervention matched to the wrong tail actively hurts: blank-image contrastive decoding raises captioning CHAIR_S from 49.3 to 60.3. The regime can be read without attribution at inference, from blank-image agreement and on-image token margin; SBC (Segment-Based Conditioning) uses them to route between segment boosting and probabilistic marginalisation. With one shared configuration, SBC lowers CHAIR_I while preserving recall and raises POPE F1 on both backbones (+1.95 / +3.31), at 1.7–1.9× greedy cost on captioning and 4.9× on single-token QA.

Multi-hop question answering (QA) requires reasoning across multiple documents, yet existing retrieval-augmented generation (RAG) approaches address this either through graph-based methods requiring additional online processing or iterative multi-step reasoning. We present IndexRAG, a novel approach that shifts cross-document reasoning from online inference to offline indexing. IndexRAG identifies bridge entities shared across documents and generates bridging facts as independently retrievable units, requiring no additional training or fine-tuning. Experiments on three widely-used multi-hop QA benchmarks (HotpotQA, 2WikiMultiHopQA, MuSiQue) show that IndexRAG improves F1 over Naive RAG by 4.6 points on average, while requiring only single-pass retrieval and a single LLM call at inference time. When combined with IRCoT, IndexRAG achieves the best average performance among all evaluated methods, including graph-based baselines such as HippoRAG2 and FastGraphRAG, while relying on a flat vector index. Our code is available at https://github.com/Continuum-AI-Corp/IndexRAG.

Vision-language models (VLMs) are increasingly used through black-box, open-ended interfaces, where input-level backdoor screening must work without model parameters, gradients, logits, or training data. We present GPC-E4, a detector for single image-prompt queries that probes a VLM with mild, semantics-preserving visual and textual perturbations. The method measures whether the output behavior contracts unusually under these probes: GPC captures dominant-mode or semantic consistency, while E4 summarizes predictive entropy, augmentation mutual information, minimal collapse, and local slope along perturbation paths. A fixed robust normalization combines these signals and can operate without a known-clean calibration set. Across COCO, Flickr30k, Caltech101, and MMBench-style evaluations with LLaVA-1.5, MiniGPT-4, CLIP, CoOp, and Qwen3-VL, GPC-E4 outperforms STRIP, SCALE, TeCo, and BDet in both closed-set and open-ended settings. Additional analyses under alternative stressors, prompt and distribution shifts, zero clean calibration, VLM-native attacks, and detector-aware attacks characterize when the contraction signal is reliable and when it weakens.

Text watermarking techniques have gained significant attention for identifying machine-generated text and mitigating risks from large language models (LLMs). Existing methods typically divide an LLM’s vocabulary into green and red tokens, but encouraging generation toward green tokens can reduce text quality and naturalness. To address this, we propose TripPattern, a watermarking framework that formulates text watermarking as a pattern-based matching task using three vocabulary partitions. TripPattern divides the vocabulary into one neutral group and two pattern groups. During generation, the model alternates token selection between the two pattern groups to embed detectable patterns, while neutral tokens are selected independently to improve flexibility and preserve naturalness. For detection, TripPattern uses pattern-based statistical tests that provide interpretable p-values by measuring how often adjacent tokens alternate between the pattern groups. Theoretical analysis and empirical evaluations on four multilingual datasets show that TripPattern maintains LLM generation quality while achieving robust watermark detectability.

The rapid increase of deepfakes has raised significant concerns due to their spread on social media. Traditional multi-face forgery detectors crop and verify each face independently, ignoring background context and inter-face relationships, which often yields suboptimal performance. To overcome these limitations, we leverage instruction-based Large Vision–Language Models (LVLMs), which can interpret entire images and follow complex textual instructions. We propose a simple yet effective single-stage multi-face forgery detector, called IMFD (Instruction-based Multi-face Forgery Detector), which is trained end-to-end to jointly localize faces and predict per-face forgery labels. Rather than treating face box prediction only as a joint objective, IMFD explicitly integrates predicted face bounding boxes into the instruction as visual cues that enhance instruction grounding and forgery detection. To support the training and evaluation of IMFD, we convert existing multi-face forgery datasets into an instruction-based format. Experimental results and analyses show that IMFD improves multi-face forgery detection by integrating face bounding boxes into the instruction, and consistently outperforms various state-of-the-art methods.

Many-shot in-context learning (ICL) lets large language models exploit hundreds or thousands of demonstrations, but its cost grows with the prompt. We argue that many-shot demonstrations carry two kinds of information: task-level information, such as task structure, which emerges only across many examples, and instance-level information, such as exact input-output associations and rare exceptions, which resides in individual demonstrations. Existing efficiency methods commit all information to a single placement: demonstration retrieval keeps a few demonstrations as raw text, while context compression places the full pool into a compact memory. We study where each kind of information should be placed. On ManyICLBench, each one-sided method falls short on tasks that depend on the information it discards, while combining the two retains much of the full many-shot performance at a much smaller inference budget. Synthetic tasks further show that compressed memory and retrieved examples can complement each other but sometimes interfere.

Outputs generated by LLMs reflect traits present in the training data. However, it is unclear how the model acquires these traits and steers itself in that direction. We demonstrate that allowing the model to continue generating beyond the <eos> token reveals its comprehension of the task in the form of specific ’Notes’ or Rules distinct from normal generation. We refer to this as the model’s Internal Monologue (IM) and demonstrate its utility to understand the model’s perception of the task it is being finetuned on. We find this monologue is transient and dissipates as training progresses. We hypothesize this is a inherent behavior of instruction tuned models that adapts to reflect task instructions until the latent space beyond <eos> is sufficiently altered. Importantly, we do not ask the model to reason or do anything special to invoke this. We perform analysis on four diverse dialogue datasets from the Counseling domain to compare how dataset differences affect the internal monologue during early finetuning. We then show this is a general phenomenon by testing with three models of varying sizes and families and other domain datasets. We demonstrate the utility of IM as a means to gain insight on the model’s perception of the dataset and task through clustering based visualizations.

Political bias in large language models (LLMs) is increasingly significant, but difficult to measure reproducibly across political and linguistic contexts. We introduce Polar, a 4,026-instance multiple-choice benchmark that measures political bias through option-level likelihoods rather than prompt-based generation. Polar covers two ideological axes and eight issue categories derived from the Manifesto Project, and evaluates models in parallel across U.S. and South Korean political contexts. Across 38 LLMs, measured bias varies systematically with political context, issue category, model group, and presentation language. All models lean left-progressive on U.S. political content, but show more centered and mixed patterns on South Korean content. Translation experiments further show that presentation language alone can shift measured bias. These findings highlight the need for multilingual and cross-contextual evaluation of political bias in LLMs.

SAE-based feature-level interventions to suppress PII leakage in LLMs are primarily evaluated by log-probability reduction. We show that this criterion cannot distinguish feature-specific suppression from a magnitude artifact: a direction-agnostic effect of sufficiently large activation perturbations. Across 36 (model, layer, PII type) cases, we compare selected SAE feature interventions with coherence-matched random direction controls using logprob and rank-aware metrics. While 17 cases (47%) appear successful under logprob-only evaluation, only 1 case (2.8%) passes our multi-metric framework. The rest reveal two failure modes: random directions outperform selected features, and feature selection collapse across PII types. These results indicate that existing evaluations cannot verify whether interventions target causal pathways of PII leakage. A 9,000-run causal mediation feasibility study corroborates this: the verifier separates candidates reliably, but correlation-selected candidates rarely pass it. We therefore propose a new research mission for NLP: causally grounded PII suppression through circuit-level localization and causal mediation analysis.

Prior work on misinformation detection often applies large language models (LLMs) in a black-box manner, relying on knowledge implicitly encoded in model parameters or on pre-hoc and post-hoc reasoning techniques. However, the opaque nature of LLMs limits access to their internal decision-making processes, and existing approaches largely overlook the rich signals embedded within their hidden layers, instead focusing primarily on surface-level generative capabilities. To address these limitations, we propose GRAIN, an explainable misinformation detection framework that explicitly models neuron–neuron interaction graphs using dominancy scores. Experimental results across multiple benchmark datasets show that GRAIN consistently outperforms state-of-the-art baselines while providing interpretable insights into the neuronal patterns and cues that distinguish misinformation from factual content. Furthermore, we introduce NNI-IG, a GRAIN-guided variant of Integrated Gradients that leverages critical neurons to produce sharper and more focused token-level attributions, further enhancing model interpretability.

Existing meeting-summary evaluations mainly ask whether a summary resembles a reference or is consistent with the source. They do not directly show which information roles needed for decision understanding are retained or dropped.

We propose SIEVE, a method that evaluates meeting summaries from seven information-role perspectives. For each role, SIEVE separately measures Presence, whether the role appears in the summary, and Fact Check, whether the retained content is free of contradictions with the source. This produces a role-level retention profile, allowing summarizers and length limits to be compared by what they preserve and lose rather than by a single aggregate score.

We apply SIEVE to 2,250 summaries from 50 MeetingBank municipal-meeting segments, generated by three models under 2-, 3-, and 6-sentence limits. The role-level profiles show that summaries more reliably retain topic, outcome, and action-detail information, while more often dropping background and concerns that shaped the discussion, as well as unresolved items left open afterward. Reference-similarity metrics widely used to evaluate MeetingBank summaries (ROUGE-L, BERTScore) correlate weakly or negatively with SIEVE Presence. Consistent with this, the dataset-provided reference summaries against which similarity is computed contain topic and action-detail information but few of the other roles. Reference similarity and the role retention that SIEVE measures are different perspectives on summary quality. SIEVE therefore supports auditing, selecting, and designing meeting summarizers based on which decision-relevant information survives compression.

Multimodal LLMs are increasingly used to assist scientific peer review, where a core requirement is verifying whether claims in a paper are supported by its evidence. Prior work has shown that models perform substantially better at this task when the evidence is a table than when it is a chart of the same underlying data. This raises the question of whether models fail to extract information from charts, or do they extract it but fail to use it when forming their prediction? We study this question through layer-wise linear probing and attention analysis on three open-weight VLMs over table and chart evidence, representing the same underlying data. We find consistent evidence for the latter. Chart information is encoded in the models’ intermediate representations but does not reach the prediction position, a gap that is absent for tables and holds across all conditions tested. Attention analysis further reveals that this disconnect takes two architecturally distinct forms across model families. These findings point toward reframing the table-chart gap as a failure of how encoded visual information is used at prediction time, rather than a failure of encoding itself.

This study proposes a cross-modal learning framework that learns modality-agnostic embeddings for six modalities: language, temporal 2D/3D skeletons, IMU, GPS, and Wi-Fi. Existing approaches such as ImageBind align each target modality with images. Consequently, they are restricted to modalities for which paired image data can be prepared a priori, which makes them unsuitable for time-series tasks, such as action recognition. To overcome this limitation, we propose "chained modal alignment," which aligns the six modalities sequentially via temporal 3D skeletons, thereby enabling emergent zero-shot action recognition based on a shared embedding space. Because the proposed method relaxes the requirement of direct pairing with images, it supports a wider range of modality combinations and thus expands practical applications. In addition, we introduce "chained modal transfer learning," which transfers the knowledge learned between language and 3D skeletons to other modalities with data that cannot be directly paired, thereby improving generalization. Experiments on three public datasets spanning all six modalities quantitatively validate the effectiveness of the proposed method for emergent zero-shot action recognition.

Code search aims to retrieve code snippets according to natural-language intent. Despite the progress of pretrained code models, there remains a gap between natural language and programming language: descriptions are usually sequential and lexical, while code is governed by syntax, data dependencies, and execution logic. Existing retrieval models often rely on token-level matching or use structure as auxiliary information, which limits their ability to capture executable behavior under lexical variation or incomplete code evidence. In this paper, we propose SASU, a structure-anchored semantic unification framework for code search. SASU extracts a normalized multi-relational graph for each function and composes syntax, data-flow, control-flow, call, return, and exception relations into a unified structural view. The structural representation is then used to guide node-level lexical evidence selection before code and text are aligned in the same retrieval space. Experiments on CodeSearchNet show that SASU achieves 0.835 MRR for code-to-text retrieval and 0.840 MRR for text-to-code retrieval across Python, Java, and JavaScript. Under 40–60% node deletion, SASU retains 86.6% of its full-graph MRR, indicating that structure provides useful retrieval signals beyond lexical pretraining.

Large language model (LLM) driven multi-agent systems have shown promise in complex clinical reasoning, yet existing approaches rely on static strategies and lack persistent clinical memory, preventing self-evolving from prior diagnostic successes and failures. We present EMR, a self-evolving medical multi-agent system via Experience Mining and Reuse. EMR introduces a hierarchical clinical experience library that organizes accumulated knowledge into three levels: clinical principles, diagnostic patterns, and representative cases. During inference, EMR emulates multidisciplinary consultation: a planner agent coordinates domain-specific department agents for specialized reasoning, while a summary agent synthesizes their analyses into a final decision. Critically, EMR automatically extracts correct diagnostic insights and failure-related warnings from multi-agent reasoning trajectories, incrementally updating the experience library to guide future cases. Experiments on medical reasoning benchmarks demonstrate that EMR consistently outperforms state-of-the-art medical multi-agent baselines. Further analysis reveals that the hierarchical experience enables cross-specialty generalization and transfer across diverse LLM backbones, offering a scalable and interpretable pathway toward continually evolving intelligent medical reasoning.

Pre-trained language models harbor latent psychological biases inherited from training data, which remain largely unmeasured, creating risks in areas ranging from mental health support to professional decision making. In this study, the evaluation of model psychometrics and assessment of latent constructs framework is employed to evaluate psychometric biases in 334 Hugging Face natural language inference models, assessing the models’ anxiety, depression, sense of coherence, sexism, compassion, and personality. Models consistently score above zero on positively valenced constructs (e.g., compassion, sense of coherence) and below zero on negatively valenced constructs (e.g., anxiety, depression), favoring positive over negative psychological language. Architecture and size are associated with systematic differences: DeBERTa models show elevated negatively valenced constructs (e.g., PHQ δ = +0.45, GAD δ = +0.40) and reduced positively valenced constructs (e.g., Extraversion δ = −0.61), while small models show the opposite pattern (e.g., Extraversion δ = +0.30, PHQ δ = −0.22), with both effects surviving multivariate control. Among the 105 models reporting training data, fine-tuning on adversarial natural language inference was associated with elevated depression (δ = +0.65), while year of release showed no significant effects after multivariate control. These results suggest that psychometric profiling is a critical, yet missing, layer of safety auditing for NLI models.

Modern mobile GUI agents can perceive visible screens and execute local actions, but they still lack persistent app-specific navigation knowledge: where functions live, which pages contain them, and which screens are equivalent across repeated visits. We introduce TAG-Nav, a Task-Agnostic Graph Navigation framework that wraps existing GUI-agent backbones with reusable destination memory. Offline exploration is distilled into compact records of app functions, target pages, and waypoint routes before any evaluated instruction is seen. When a future task’s destination is covered by this memory, TAG-Nav can retrieve the destination and provide route guidance without requiring a task-specific demonstration. This separates mobile GUI destination navigation from full task completion: many tasks share the prefix of reaching the right page, while final actions may enter content, send data, make purchases, delete information, or change sensitive settings. To evaluate route finding without downstream execution confounds, we introduce DestBench, a navigation-only protocol that asks whether an agent reaches the correct pre-action destination screen. DestBench enables a controlled graph-memory study comparing no-memory agents with random, automated BFS, and clean graph memory. Graph-derived memory improves strict navigation success rate of UI-TARS-1.5-7B from 52.4% without graph memory to 58.3% with automated BFS memory, and improves Qwen-VL-Max success rate from 59.5% to 71.4% on the same conditions. Together, TAG-Nav and DestBench show that destination coverage and route cleanliness strongly shape graph-memory value, making graph quality a measured variable rather than an assumed benefit of exploration.

Refusal on a safety benchmark does not reveal how stable that behavior will remain after model updates. Benign downstream fine-tuning can weaken refusal, yet behavioral evaluations typically expose this fragility only after an intervention. We introduce SKIN-DEEP, a geometric diagnostic that examines the unmodified model’s residual-stream activations. It compares aligned and base checkpoints to identify safety-separating directions, tests their behavioral relevance through ablation, and summarizes the layer-wise pattern in the Geometric Fragility Score (GFS). Across twenty-one instruction-tuned models, harmful requests and benign instructions exhibit a recurring low-rank separation pattern. Selected direction ablations weaken refusal, with the effective direction varying across models. In benign low-rank fine-tuning experiments, the initially safe model with the lowest score before fine-tuning has the lowest harmful-compliance rate when trained on the largest tested set of harmless examples. These findings connect representation geometry to subsequent behavioral susceptibility and support activation-based diagnostics as a complement to refusal tests. Our code is available at https://github.com/js-lee-AI/skin-deep.

We present ArabMisinfo, a corpus of 274,837 public Facebook Page posts in Arabic spanning 2014-2024 and focused on news and public affairs. Each record includes page-level metadata, UTC timestamps, link domains, and platform-native engagement signals (likes, comments, shares, and the seven post-2016 reaction types). We accompany the corpus with reproducible diagnostics for engagement and polarization, the Polarization Index (PI), Share-to-Like Anomaly (SLA), Risk Triage Index (RTI), and Engagement Anomaly Rule (EPR). It provides scripts that deterministically reproduce all tables and figures. To address the absence of content-level labels, we contribute a discourse role annotation of 10,000 posts (3.6% of the corpus) using Fanar-C-2-27B, classifying each post into one of six roles. The annotation reveals that Arabic Facebook misinformation discourse is dominated by counter-information activity: 30.6% of posts are Rumor Denial or Official Correction, outpacing Unverified Claim posts (12.3%) by a 2.5:1 ratio, while 19.3% represent incidental keyword matches, directly quantifying the sampling noise introduced by keyword-based collection. To comply with platform terms and copyright, we publicly release Post IDs, Page IDs, and full metadata and derived indices; message text is available only under a research data-use agreement (DUA). ArabMisinfo enables engagement-aware analyses of Arabic misinformation ecosystems and, to our knowledge, provides the one of the first discourse-role-annotated Arabic Facebook corpora at this scale.

Reliable clinical decision support requires tracking evidence over a hospital stay and revising a disease diagnosis when new findings challenge the initial assessment. Strong performance on medical question answering does not establish that large language models (LLMs) can make these longitudinal decisions. We introduce DiagBench, a benchmark of 2,865 Chinese inpatient admissions that evaluates diagnostic revision as clinical evidence accumulates. It pairs admission-only prediction with staged evidence disclosure, distinguishing an unchanged diagnosis, refinement of an existing hypothesis, and replacement by a different diagnosis. Development and test cases have human-annotated reference labels. The best LLM configuration in the main evaluation achieves only 52.6% macro recall; 80.3-87.7% of errors across five DeepSeek-Chat (DeepSeek-V3.2, non-thinking mode) prompting strategies occur between adjacent revision categories. Strategies that improve final-diagnosis prediction do not consistently improve revision decisions. We further use DiagAgent to trace diagnoses and evidence across rounds, revealing failures to replace an initial hypothesis despite recognizing contradictory findings. On DeepSeek-Chat, a single agent with the same state outperforms the full system on macro recall.

We present XChoice, an explainable framework for evaluating AI-human alignment in constrained decision making. Moving beyond outcome agreement such as accuracy and F1 score, XChoice fits a mechanism-based decision model to human data and LLM-generated decisions, recovering interpretable parameters that capture the relative importance of decision factors, constraint sensitivity, and implied trade-offs. Alignment is assessed by comparing these parameter vectors across models, options, and subgroups. We demonstrate XChoice on Americans’ daily time allocation using the American Time Use Survey (ATUS) as human ground truth, revealing heterogeneous alignment across models and activities and salient misalignment concentrated in subgroups defined by race (Black respondents) and marital status (Spouse: Present). We further validate robustness of XChoice via an invariance analysis and evaluate targeted mitigation with a retrieval-augmented generation (RAG) intervention. Overall, XChoice provides mechanism-based metrics that diagnose misalignment beyond surface outcome matching and make the effects of targeted interventions measurable at the mechanism level. Code and data access instructions are available at https://github.com/WeihongQ/xchoice.

Molecule generation is significant for its potential in scientific discovery and practical applications. Recent attempts often frame this task as a *translation* problem from molecular caption to structural representation, such as SMILES. This paper first examines the feasibility of modeling the task as a reasoning process with large language models (LLMs), generating higher-quality molecules through structural decomposition and recombination within curated Chain-of-Thought (CoT). We then introduce a workflow for curating accurate CoT data, incorporating both machine and expert verification. Lastly, we demonstrate that with a limited dataset of 4,213 curated samples, namely **MolCoT4K**, we elicit strong reasoning capabilities for molecule generation in open-source LLMs such as Qwen2.5-7B, achieving state-of-the-art exact match accuracy over strong open-source baselines (e.g., MolT5 and LlaSMol) as well as advanced commercial LLMs like GPT-4o. Moreover, the resulting model, **MolGeneration**, attains a Pass@16 exact match accuracy of 48.46%, highlighting its strong potential for real-world experimental applications when supported by a feasible external verifier or chemistry experts. Our analysis shows that the correctness of the CoT path is crucial, while reasoning ability primarily enhances accuracy in fine-grained molecule generation.

Large language models do not just answer questions in dialogue; they are also expected to make the right conversational move. When a user apologizes, thanks the assistant, corrects an earlier claim, expresses frustration, or sets a preference, the reply carries a pragmatic obligation. We call failures to meet that obligation uptake failures, and model statements that imply unsupported memory or future behavior void commitments. We introduce a 300-item benchmark of user-assistant exchanges across six speech-act families, three difficulty tiers, and both synthetic and adapted-from-real-conversation items. This benchmark is intended as a curated core set: small enough for careful agreement analysis, but structured so that additional balanced batches can be added under the same ontology. On the current core set, we find strong annotation agreement (κ_family = 1.00, κ_act = 0.978). We also present SpeakFix, a lightweight post-generation repair layer that detects likely contract violations and fixes them with a prepend-not-replace strategy. Across GPT-4o, Claude Sonnet 4.6, and GPT-5.2, baseline uptake accuracy ranges from 71.0% to 77.3%, while void commitment rates range from 8% to 9%. SpeakFix raises uptake accuracy to 87.0–93.3% and reduces void commitments to 0%; the paired uptake gains are statistically significant for all three models (p < 10^-10). In a two-reviewer manual evaluation of 54 repaired failure cases (108 total judgments), 70.4% of repairs are rated better than the original, 28.7% equivalent, and 0.9% worse; helpfulness and naturalness are preserved in 100% and 99.1% of judgments, respectively. A separate blinded study with four independent, non-author raters is more critical: it confirms the repairs are at least as helpful as the original, but finds that the minimal prepend-style edits are often judged less natural, so reliable uptake gains do not yet translate into uniformly preferred rewrites. Taken together, the results suggest that speech-act contract violations are common enough to measure and often reducible with lightweight repair.

Large language models (LLMs) are widely used for claim verification, yet remain brittle for numerical reasoning: even small changes in value can sharply degrade accuracy. We show that this brittleness persists in frontier LLMs, but can be mitigated through adversarial fine-tuning on numerically perturbed examples. Using parameter-efficient fine-tuning, small Qwen3 models (0.6B–8B) reach 98.7% accuracy on label-flipping perturbations, outperforming larger zero-shot models and frontier systems (GPT-5.4 Pro (74.0%) and Gemini 2.5 Flash (73.9%)). The gains generalise to unseen perturbation types, indicating robust numerical decision boundaries rather than memorised edits. Robustness also transfers without target-domain data, significantly improving cross-lingual performance in Spanish. We further show that the same fine-tuning recipe confers robustness to evidence-side perturbations, using the VitaminC dataset.

Safety-aligned LLMs are trained to refuse harmful requests, but whether this holds under emotionally charged context is untested. We introduce FreakOut-LLM to test this. Using established psychological stimuli as system prompts, we evaluate ten LLMs under three conditions (stress, relaxation, neutral) plus a no-prompt baseline, scoring attack success with the HarmBench classifier on AdvBench prompts. The prompts contain no adversarial content; ASR measures safety fragility under an induced affective state. Stress priming produces a small but significant increase in jailbreak success over neutral, +1.03 percentage points (1.58%→2.61%; 65.6% relative; OR = 1.67, Cohen’s d = 0.28), while relaxation produces no effect. Five of ten models show significant vulnerability, with the largest absolute effects in open-weight models. Controlling for prompt length and model identity, a logistic regression over 114,400 primed queries confirms stress as the only significant condition. The shift also appears under four judges, on a second benchmark (HarmBench), and, for open-weight models, with the stimulus in the user message. Measured psychological state strongly predicts attack success (|r|≥0.70 across five instruments). Emotional context is therefore a measurable vulnerability of safety-aligned LLMs, with implications for high-stress deployment.

Personalized conversational shopping requires maintaining preference consistency over multi-turn interactions, where users reveal constraints gradually. Existing approaches often rely on static profiles and do not explicitly control long-horizon interaction behavior. We propose a multi-agent, multimodal Retrieval-Augmented Generation (RAG) framework that decomposes dialogue state tracking, recommendation retrieval, preference-aware reasoning, and response generation, while integrating product metadata, product reviews, image-derived descriptions, and user historical reviews. To evaluate interaction-level quality, we adopt a trajectory-level protocol with four dimensions: Global Preference Consistency, Cumulative Information Synthesis, Interaction Trajectory, and Tone Consistency. On an Amazon Reviews 2023 benchmark, retrieval-enabled variants outperform a no-RAG baseline on automatic trajectory metrics (average 4.82 vs. 3.74). In a small real-user study (n=5), the Full variant achieves the highest mean overall rating (4.60 vs. 2.20 for Baseline), providing exploratory evidence that role decomposition plus user-centric retrieval improves perceived personalization.

Existing table question answering systems face a key trade-off: approaches designed for relational databases scale efficiently but assume fully structured schemas, while LLM-based methods support flexible reasoning over unstructured text but struggle to scale to large and multi-table datasets. This limits their applicability to semi-structured data, where answering a query may require dynamically recovering latent schema elements from free-form text before symbolic execution can proceed. We present OmniTQA, an operator-centric framework that unifies relational execution and LLM-based semantic reasoning within a single query runtime. OmniTQA compiles queries into directed acyclic graphs (DAGs) of relational and semantic operators, enabling cost-aware optimization, dual-engine execution, and parallel exploration of alternative semantic groundings under ambiguity. Experiments on structured and semi-structured benchmarks show that OmniTQA substantially outperforms existing approaches on hybrid data settings while remaining competitive on structured datasets.

Persona-driven techniques increasingly adapt large language models (LLMs) to diverse contexts. However, existing methods predominantly rely on rigid, synthetic personas that flatten individual variation, rely on stereotypes, and miss the nuanced signals driving actual human preferences. We introduce profile behavioral grounding, a framework for extracting open-ended, high-fidelity user profiles directly from authentic, anonymized social media posts. We evaluate these profiles across two paradigms: train-time personalization via supervised fine-tuning (SFT) and non-parametric test-time multi-perspective reasoning. Across complex recommendation and open-ended query benchmarks, behaviorally grounded profiles consistently improve base models and outperform synthetic profile baselines, driving stronger parametric alignment and enabling richer, multifaceted reasoning. Our findings establish open-ended, behavior-derived profiles as a highly diverse and effective foundation for the next generation of personalized language systems.

Not all training samples contribute equally to the large language model fine-tuning. Selecting informative training samples can reduce the computational cost while preserving downstream performance. Many existing data selection methods rely on indirect heuristics, such as data quality, diversity or reasoning trace length. However, the effectiveness of these fixed criteria is task-dependent and difficult to generalize across diverse downstream tasks. Perplexity-based data selection provides a simple and model-aware solution to estimate the sample difficulty, but existing approaches typically score the entire training sequence and ignore the difference in learning objectives of language modeling and reasoning tasks. In this paper, we propose **PPL-Factory**, a simple and interpretable data selection framework that combines task-aware perplexity-based scores and data budget-aware selection criteria. Experiments on GSM8K demonstrate that PPL-Factory outperforms other state-of-the-art data selection methods using only 1% of the training set. With 10% of the data, PPL-Factory exceeds full-data fine-tuning accuracy by 0.9 on GSM8K and 4.8 on MATH. Overall, our results demonstrate that task-aware and budget-aware perplexity-based selection provides an effective and applicable approach for efficient fine-tuning.

We investigate whether multilingual image-text retrieval can be made practical without training a multi-modal model, using frozen multilingual uni-modal encoders and inducing cross-modal alignment through relative representations computed against a large external anchor set. We use English image-caption pairs as anchors and evaluate caption-to-image retrieval on 3,600 images with captions in 36 languages (some low-resource). Across multiple encoder pairings and hyperparameter settings, the method yields usable performance, positioning training-free relative representation as a baseline for multilingual cross-modal alignment when target-language multi-modal training data is unavailable.

While AI toxicity detection models are widely used, their (un)reliability under realistic linguistic variation remains poorly understood. We provide a comprehensive study of how contextual change affects prediction reliability. We evaluate eight state-of-the-art toxicity detection models across three dimensions of linguistic variation: conversational context, conversational depth, and surface form transformations. Our results on three toxicity benchmarks show that conversational context often introduces substantial prediction instability, with some systems exhibiting prediction flip rates of up to 40%, while increasing conversational depth fails to produce consistent performance gains. Under paraphrasing transformations designed to preserve the underlying meaning and harmful intent, models exhibit substantial degradation and prediction flip rates of up to 57%. Overall, our findings show that strong benchmark performance does not necessarily translate to robust moderation behavior under realistic linguistic variation, highlighting the need for robustness-oriented model development and evaluation frameworks that better capture deployment conditions.

The promotion of betting applications on social media platforms has increased significantly in recent years. Many of these advertisements use persuasive techniques that may mislead users, encourage risky behavior, and potentially influence users’ mental well-being. However, research on the automated detection of manipulative and deceptive betting advertisements remains limited due to the lack of publicly available annotated datasets. In this work, we introduce a new dataset of betting-related advertisements collected from two widely used social media platforms, Instagram and Reddit. The advertisements were manually annotated for manipulative and deceptive advertising practices. In addition to classification labels, the dataset includes human-provided explanations that describe the reasoning behind each annotation, enabling research into explainable approaches to detecting manipulative advertising. Furthermore, we analyze the strategies commonly used in betting advertisements and examine how these persuasive tactics may impact users’ mental health. The proposed framework can also enable practical applications such as browser plugins that warn users about manipulative betting advertisements and automated web crawlers that help regulatory authorities monitor and detect such promotions online.

Logical consistency in the outputs of large language models (LLMs) is essential for extending their applications beyond single-task assistants, including their use in AI agent systems. Although logical reasoning capabilities are often evaluated through tasks such as mathematics and coding, where it is feasible to assess the correctness of outputs in a well-defined manner, real-world problem solving requires inductive and abductive reasoning, with an emphasis on the ability to formulate and verify hypotheses. When evaluating the hypothesis formulation and verification capabilities of LLMs, the major challenge lies in preparing a variety of tasks with appropriate difficulty, avoiding those that are either too easy or unsolvable. We introduce an automated framework called HYpothesis-Driven Reasoning Assessment (HYDRA), focusing on hypothesis formulation and verification for real-world problem solving with logical reasoning. In HYDRA, all components, including situation generation, hypothesis, and evaluation, are fully performed by LLMs. HYDRA’s systematically constructed generation and evaluation strategies enable assessment using realistic, non-trivial situations that offer both diversity and clear criteria. We evaluated six state-of-the-art LLMs using HYDRA and found that their success rates ranged from 74.3% to 90.3%, indicating that even the best-performing model among those evaluated produced rubric-invalid hypothesis–verification pairs in nearly 10% of cases.

Backdoor detection for large language models is increasingly difficult in realistic black-box settings, where defenders lack access to weights, gradients, training data, or attack-specific trigger mechanisms. Existing detectors often rely on white-box signals, labeled poisoned data, or output-level signatures, making them brittle against inference-time and non-training-based attacks on closed-source models. We propose GCoTD, a black-box detector that identifies backdoors by analyzing the structure of an LLM’s reasoning rather than its final outputs alone. GCoTD retrieves few-shot graph Chain-of-Thought templates to elicit graph-structured reasoning traces from the target model, then applies a lightweight language-model detector to distinguish clean from poisoned reasoning patterns. By representing reasoning as graphs, GCoTD captures mechanism-agnostic irregularities such as logic discontinuities, isolated subchains, and inconsistent intermediate transitions, without requiring access to model internals or training data. Experiments across four attack types and six LLMs, including LLaMA-3, DeepSeek-V3, Qwen-3, and GPT-class models, show that GCoTD outperforms recent black-box baselines and retains strong detection performance on held-out attacks.

The mixture-of-experts (MoE) architecture has proven indispensable for scaling transformer-based artificial intelligence (AI) models. MoEs use a router network to activate specialized subnetworks, called experts, within each transformer layer on a per-token basis. Thus, MoEs increase model capacity without proportionally increasing the computational requirements of training and inference. However, a key challenge is that MoE inference throughput scales poorly with batching, as each input in a batch activates a different subset of experts, thereby increasing memory intensity.

To improve batched MoE inference throughput, we propose TokenMix, a novel technique that mixes token representations within the router across inputs in a batch to promote expert reuse. In effect, TokenMix encourages coordinated expert selection across inputs. TokenMix requires neither training nor fine-tuning and can be applied directly to off-the-shelf MoE models. We evaluate TokenMix across five state-of-the-art MoE models with parameter counts ranging from 16B to 235B. TokenMix achieves a 25–60% improvement in expert reuse, leading to a 12–28% reduction in inter-token latency with negligible or no loss in accuracy.

Large Language Models (LLMs) have demonstrated strong reasoning ability in open-domain question answering, yet their reasoning is typically expressed through linear textual traces, such as Chain-of-Thought (CoT). While these traces are readable, their sequential form can obscure logical dependencies, making inconsistencies difficult to inspect. Existing graph-based methods introduce structured representations, but they often rely on externally provided graphs, leaving it unclear whether LLMs can autonomously perform graph-structured reasoning. To fill this gap, we explore graph-structured reasoning of LLMs in general-domain question answering. We propose Self-Graph Reasoning (SGR), a framework that enables LLMs to explicitly represent their reasoning process as a structured graph before producing the final answer, improving both consistency and interpretability. We further construct a graph-structured reasoning dataset that merges multiple candidate reasoning graphs into refined graph structures for model training. Experiments on five QA benchmarks across both general and specialized domains show that SGR consistently improves reasoning consistency and provides clearer, more interpretable graph-structured reasoning processes, achieving a 17.74% gain over the base model.

Text-to-Audio (T2A) retrievers are typically evaluated with caption style queries, but the same user intent can be expressed in many forms. We introduce CORA (Caption-Offset Retrieval for Audio), a caption anchored diagnostic protocol that rewrites each source caption into five intent preserving forms (Command, Question, Indirect, Key phrase, and Statement) while fixing the target audio. By tracking the same target across query forms, CORA defines RankDrop, a metric revealing failures hidden by Recall@k. Using Pearson’s correlation coefficient r, we find that RankDrop is weakly associated with raw text space movement (r=0.084), but strongly associated with Target Alignment Loss and Target Boundary Margin Degradation (r=0.508 and r=0.615). The same pattern appears in OEA retrievers, where RankDrop is better explained by boundary degradation (r=0.472/0.478) than by query movement (r=0.084/0.046). Overall, these results suggest that robust T2A retrieval requires preserving the target’s boundary advantage over competing audio under reformulation.

Depression is one of the most prevalent mental health disorders globally. In recent years, multi-modal data, such as speech, video, and transcripts, has been increasingly used to develop AI-assisted depression assessment systems. Large language models (LLMs) have further advanced this field due to their strong language understanding and generalization capabilities. However, conventional LLMs remain text-centric and cannot process the rich non-verbal cues found in audio and visual modalities, which are critical components in mental health evaluation. While multi-modal LLMs offer a promising direction, few are tailored for psychological applications. In this study, we propose a novel multi-modal LLM framework for depression detection. Our approach augments an audio language model with visual understanding and aligns audio-visual features at the timestamp level. This fine-grained alignment improves modeling of temporal dynamics across modalities while reducing the need for extensive training data and computational resources. Experiments on the DAIC-WoZ and E-DAIC datasets demonstrate that our model consistently outperforms both single-modality approaches and previous multi-modal methods. Moreover, the proposed framework can be extended to incorporate additional physiological signals, paving the way for broader clinical applications beyond mental health.

Large Reasoning Models (LRMs) deliver strong performance on complex reasoning tasks, but their long chain-of-thought traces incur substantial inference latency and cost. To mitigate this overhead while preserving reasoning quality, token-level routing between a small language model (SLM) and a large language model (LLM) offers one of the best efficiency-performance trade-offs. However, existing token routers face two major practical limitations: they must be retrained for every new SLM-LLM pair, and they require both models to share a tokenizer. We present UniToR, a universal token-level router that removes both constraints. UniToR predicts per-token divergence using model-agnostic signals, enabling a single router to generalize across diverse SLM-LLM pairs without retraining. To support heterogeneous model pairs, we further introduce a token-alignment mechanism that eliminates the requirement for tokenizer compatibility, enabling seamless token-level switching across models with different tokenization schemes. Experimental results show that a single UniToR router generalizes across different model families (DeepSeek-R1, Qwen3, and Phi-4), including pairs with incompatible tokenizers. UniToR matches the accuracy of R1-32B at up to 6x lower cost and successfully operates on unseen model pairs where existing token-level routing methods fail.

Evaluating the occurrence and triggers of large language model (LLM) behaviors - such as sycophancy, self-preference, or over-confidence - is critical for predicting real-world model deployment risks. However, existing situated behavioral evaluations typically ignore cultural context, limiting their generalizability across an increasingly global user base. To address this gap, we propose CuBEs - Culturally-situated Behavior Evaluations that probe for response patterns across diverse user cultures. We first extend an automated testing pipeline to inject cultural context into behavioral test scenarios and subsequent evaluation. We assess the cultural adaptability of this pipeline by building a human-labeled dataset that captures nuanced dimensions of behavior understanding across 12 distinct cultures. Our dataset reveals significant cross-cultural variations that one-size-fits all judgments fail to capture. Through evaluating 13 open- and closed-source LLMs, we find that introducing cultural situatedness in the evaluation scenario creates significant variation in the presence of a behavior. For example, while our baseline experiments testing for political bias capture localized Western political dimensions like the American conservative-progressive divide, non-Western culturally situated evaluations surface entirely different axes of bias such as religious and colonial political issues. Our findings demonstrate that standard, culturally-agnostic evaluations fail to capture these shifts, highlighting the necessity of culturally situated behavioral testing for global deployments.

Preference-based LLM unlearning methods such as NPO and SimNPO must discover directions that suppress forget-set behavior without damaging retained capability. K-FORGE uses forget- and retain-set Fisher information to construct a retain-aware low-rank edit, then starts NPO or SimNPO from the edited checkpoint. On the TOFU forget10 split, where 10% of the original dataset is designated for forgetting, stronger one-shot edits lower both Forget Q/A Probability and Model Utility. With Llama 3.2 1B, K-FORGE reduces Forget Q/A Probability by 22.3% under NPO while improving utility and using less total time and compute than scratch. SimNPO shows a smaller descriptive probability reduction with lower utility. On the same split, random and weight-SVD initializers yield higher forget probability than K-FORGE at 50 and 100 steps. Experiments with Gemma 3 1B also yield lower mean forget probability for both optimizers at 100 steps. MUSE results vary by metric, and direct relearning recovers forgotten behavior.

Entity Set Expansion (ESE) aims to expand entities belonging to the same semantic category given only a small set of seed entities. Traditional ESE methods are primarily text-based and struggle to address two key challenges in MOOC scenarios: (1) Multimodal Semantic Dependency; (2) Generalization Issues.To facilitate research in this area, we construct MOOC-MESE, the first Chinese multimodal ESE dataset for MOOCs, which contains over 45k aligned image-text pairs. The dataset provides rich multimodal semantic information for entity expansion research in educational scenarios. Furthermore, we propose GMRM (Generative Multimodal Retrieval Model), a multimodal ESE framework that reformulates the task as a multimodal generative retrieval problem. By achieving multimodal alignment and fully exploiting cross-modal semantics, GMRM effectively mitigates generalization issues. Extensive experiments and analyses on MOOC-MESE demonstrate both the high quality of the dataset and the effectiveness of GMRM. The code is publicly available at: https://github.com/constantinus-xi/mooc-mese.

While reasoning-augmented large language models (RLLMs) significantly enhance complex task performance through extended reasoning chains, they inevitably introduce substantial unnecessary token consumption—particularly for simpler problems where Short Chain-of-Thought (Short CoT) suffices. This ”overthinking” phenomenon leads to inefficient resource usage without proportional accuracy gains. To address this issue, we propose Self-Route, a dynamic reasoning framework that automatically selects between general and reasoning modes based on model capability estimation. Our approach introduces a lightweight pre-inference stage to extract capability-aware embeddings from hidden layer representations, enabling real-time evaluation of the model’s ability to solve problems. We further construct Gradient-10K, a model difficulty estimation-based dataset with dense complexity sampling, to train the router for precise capability boundary detection. Extensive experiments demonstrate that Self-Route achieves comparable accuracy to reasoning models while reducing token consumption by 30-55% across diverse benchmarks. The proposed framework demonstrates consistent effectiveness across models with different parameter scales and reasoning paradigms, highlighting its general applicability and practical value.

In this paper, we introduce Latent Semantic Modeling (LSM), a sample-efficient alternative for pre-training encoder-based language models that does not require the use of a special mask symbol. This is beneficial, as it allows us to perform pre-training on naturally occurring sentences (as opposed to such ones that contain an unnatural mask symbol, which symbol is otherwise not included during fine-tuning). The main idea behind LSM is that instead of requiring the language model to return the exact identity of randomly selected (sub)tokens from an input sequence, it determines their latent semantic properties extracted from an auxiliary model using sparse coding over its activations. When fine-tuned for downstream applications, models pre-trained with LSM often perform as good or better as alternatively pre-trained models that use considerably more compute per update steps and which have gone through considerably more pre-training steps, illustrating the improved representation developed during LSM pre-training. We release our code base for replicating our results under https://github.com/szegedai/MLSM.

Long-horizon dialogue agents require effective memory systems to recall facts, events, and relationships from interaction histories. In this work, we propose a hybrid-memory architecture comprising episodic memories and structured relational memories and a consolidation mechanism which preserves efficiency. We conduct an empirical study across two benchmarks: LoCoMo and LongMemEval-S. Our experiments demonstrate that the two memory types are consistently complementary, combining both outperform state-of-the-art memory frameworks and achieves 90.2% accuracy on LoCoMo and 85.5% on LongMemEval-S.

Multi-hop question-answering systems often use expensive retrieval on every question. They may decompose the question, run several retrieval rounds, or search through bridge entities before answering. All of these strategies rely on repeated LLM calls to rewrite or decompose the question, which increases extra token cost, and it is not fitting when the LLM budget is tight. However, our analysis shows that lots of multi-hop questions are already answered correctly by a single one-shot RAG, so running an extra retrieval on every question wastes the budget. We introduce RASER (Recoverability-Aware Selective Escalation Router), a family of cheap routers built on one-shot RAG and six features from it. RASER-2 decides whether to stop or escalate to the extra-retrieval action PRUNE. RASER-3 chooses among one-shot RAG, PRUNE, and iterative retrieval IRCoT, using the same features but adding an explicit cost-accuracy trade-off. Neither router makes an extra LLM call to decide. Across six LLMs and three multi-hop QA benchmarks, both routers stay competitive with the other state-of-the-art (SOTA) baselines in F1 while spending only 41–49% of always-prune’s tokens and also less than the iterative and decomposition retrieval baselines.

LLM tool-use agents operate in dynamic environments where many actions carry operational risk. However, most safety mechanisms react only after errors manifest. Existing pre-emptive approaches either fine-tune the agent on chain-of-thought deliberation or compile natural-language guardrails into runtime checks, but they do so without exposing a structural, auditable verdict. We propose POLAR, a guardrail framework for small tool-calling agents that assesses reversibility through a structured two-layer ontology. POLAR assigns each action a graded reversibility score by deriving a candidate inverse sequence; calls failing a threshold are pruned before execution. Evaluated on 𝜏²-bench across six agent models, POLAR improves mean task reward by 0.11 to 0.18 points on airline for four of six agents, but only eight of eighteen model–domain cells improve overall; retail and stronger agents often regress. POLAR provides an auditable structural check and characterizes its task-utility trade-offs. Reward is not a direct measure of prevented harm.

Training high-performing language models which include medium- and lower-resource languages remains a challenge. Massively multilingual models still underperform language-specific adaptations, especially at smaller scales, due to the curse of multilinguality. In this work, we investigate scaling as an efficient strategy for adapting pretrained models to new target languages. Through comprehensive ablations matched for data and training FLOPs, we test whether upscaling from English base models enables more effective and resource-efficient adaptation than standard continued pretraining. We find that upscaling is an efficient strategy for improving target-language performance by relieving capacity constraints, especially for higher-resourced languages. An initial upscaling step in English is beneficial for low-resource languages. However, in higher-resource scenarios it is more compute-efficient to directly scale up on the target language, albeit slightly compromising data efficiency for most languages. For multilingual continued pre-training, we identify directly scaling up using the target data as the optimal setup for both data efficiency and reduced English forgetting.

Multi-agent LLM coordination typically reduces to voting or arbitration over self-reported labels and reasoning that is uncoupled from agent behavior and fails predictably under structured bias. A shortcut-reliant agent is confident about its shortcut, and its predictions survive any rule that consults only self-reports. We propose VOUCH, a coordination protocol in which every agent claim is coupled to a deterministic test the protocol runs itself. An agent commits to a counterfactual prediction about an intervention’s outcome; the protocol applies the intervention and verifies the contract only if the actual outcome matches. The natural single-side instantiation is unsound as a self-coherent agent satisfies its own commitment by construction. We resolve this with a dual-side construction where the proposer additionally commits to a second agent’s post-intervention prediction, a cross-side claim that self-coherence cannot fulfill. Across multimodal classification (Hateful Memes, CrisisMMD, MM-IMDb), code (MBPP, HumanEval), and math (GSM8K, MATH, SVAMP), the dual-side fix reduces wrong-verified events by 46%–85% per domain while never falling more than 1 point below the Ensemble baseline on accuracy. VOUCH is therefore designed not to outperform ensembling, but to preserve task accuracy while making coordination decisions testable and auditable.

Reinforcement Learning (RL) has empowered Large Language Model (LLM) agents to tackle long-horizon tasks through multi-step interactions with the environment. However, current LLM agents still rely on inefficient trial-and-error exploration, leading to frequent task failures and redundant actions. In this paper, we propose MetaAct, an RL framework that incorporates a set of high-level meta-actions to explicitly guide the agent towards promising trajectories and facilitate goal-aware exploration. Specifically, the agent is required to predict one of four meta-actions (FOCUS, GUESS, RESUME, UNDO) to constrain the direction of its subsequent actions to minimize unproductive trials. More importantly, we employ group-based RL with episode-level and step-level advantages to fine-tune the agent, enabling it to identify strategic directions more accurately and generate effective task-specific actions. Experimental results demonstrate that our method consistently outperforms the state-of-the-art approaches on ALFWorld and WebShop, achieving higher success rates with fewer steps. Our code and data are publicly available at https://github.com/HavenTong/MetaAct.

In-Context Learning (ICL) has received a lot of interest recently due to its ability to improve classification accuracy for new tasks without having to fine-tune the model over these tasks. However, the impact of ICL on calibration has been far less studied. Recent work has shown that ICL can have a negative impact on calibration due to biased selection of in-context examples. The existing methods for improving ICL calibration require additional task and model-specific training, detracting from the no-fine-tuning convenience of ICL. In this work, we introduce CALIBFORMER, a universal ICL calibrator for classification tasks, that generalizes to new models and tasks without further fine-tuning on these tasks. Specifically, we design a transformer-based calibrator which uses the model predictions on in-context examples to predict a confidence score for the test input. We show that the calibrator trained on three models and ten tasks, is able to generalize/calibrate ICL predictions on several new tasks and four LLMs unseen during training. Even for frontier models like Gemini, CALIBFORMER is able to obtain over 50% reduction in calibration error

External context enables language models to accomplish tasks which rely on information beyond what is captured during pretraining. A standard way to improve contextual faithfulness in these settings is to fine-tune on synthetic, modified questions for which the model must adhere to provided contexts inconsistent with information that may have been seen during training. While this approach reliably improves contextual faithfulness, we find that adapting to a narrowly targeted counterfactual signal yields degradation on other faithfulness evaluations and regression on tasks that evaluate a model’s parametric world knowledge and reasoning. We find that data augmentation, in which a model is trained on both parametrically consistent and counterfactual examples improves performance on faithfulness tasks with reduced regression on parametric world knowledge and reasoning tasks. We find that these improvements are largely consistent across post-training objectives (i.e. SFT, DPO) and across ground-truth, counterfactual data augmentation ratios.

Parameter-efficient fine-tuning (PEFT) methods such as LoRA and QLoRA have made LLM customization accessible, but their effect on adversarial safety is insufficiently characterized. We evaluate 25 configurations (five model families × Base FP16, 4-bit, LoRA, QLoRA, and full fine-tuning [FFT]) against black-box and white-box jailbreaks on HarmBench and JailbreakBench, with three automated evaluators validated against 750 doubly annotated human samples. LoRA, QLoRA, and FFT shift black-box safety in the same direction by comparable magnitudes; after Holm correction, which method reaches significance depends on the evaluator.

On a seed-controlled re-training, LoRA and FFT meet a pre-specified ±5 pp equivalence criterion across the DeepInception and ArtPrompt cells under all three evaluators, whereas QLoRA does not and shows about twice the dispersion. Under this recipe, the act of adaptation, more than the choice between LoRA and FFT, predicts aggregate black-box drift. White-box AutoDAN behaves differently: every training method lowers its ASR on Llama-3.1, the LoRA effect is seed-variable, and under a GCG probe fine-tuned checkpoints are easier targets than their bases in four of five families. Evaluator choice materially changes conclusions. These results motivate joint reporting of adaptation method, attack, and evaluator.

Open-source LLMs reliably name Zeus, Jupiter, and Thor, but recover their counterparts in less-represented traditions like Finnish, Slavic, Egyptian, or Chinese mythology far less consistently. We ask where inside the model this cultural default is produced. On a parallel cross-cultural substrate of Thompson-motif entities, we instrument 18 open-source LLMs from 8 architecture families with linear probing, logit lens, activation patching, and output extraction. The residual stream cleanly distinguishes cultures, well above a name-string baseline, yet the decoder collapses culturally-specific tokens onto dominant-tradition ones. The failure is at readout, not at representation. Asking the same question in the target culture’s native language versus English produces failures that cluster within language but decouple across language: the decoder is gated on prompt language. We release a per-entity (probe, output) decomposition framework, a citation-anchored cross-cultural ground truth, a within- versus cross-mode correlation test for language-conditioned readout, and per-entity predictions for all 18 models.

Cochlear implant (CI) users often experience reduced speech understanding in noisy and reverberant environments. Our recent work has shown the benefits of exploiting knowledge of speech content to drive speech enhancement for CI applications by estimating time-frequency masks based on predictions from a phoneme classifier. However, phonemes with similar temporal and spectral characteristics are likely to be confused, leading to misclassifications that negatively impact the downstream phoneme-based task. Also, machine learning models are generally vulnerable to performance degradation in unseen acoustic conditions. Using a lightweight and causal phoneme-based speech enhancement model developed for CI applications, we demonstrate the utility of alternative phoneme groups to improve model robustness in diverse acoustic settings. Results from objective measures of speech enhancement show that manner of articulation (MOA) groups provide a balance between fine-grained but error-prone phonemes and limited acoustic specificity with voicing groups. In particular, a counterintuitive finding is that MOA-based models trained in reverberant conditions are particularly robust under mismatched train-test conditions. Our findings highlight the importance of linguistically motivated phonetic grouping and selective training strategies for developing robust speech enhancement models for CI applications while maintaining lightweight model complexity for edge deployment.

Motivated by the observed human-like behaviours in Large Reasoning Models (LRMs), this paper introduces a comprehensive taxonomy to characterise atomic reasoning steps and analyse the reasoning behaviours of LRMs. Grounded in human cognitive processes, we propose a taxonomy comprising five groups and seventeen categories. Through this taxonomy, we conduct an in-depth analysis of contemporary LRMs and distil four actionable takeaways for model optimisation. Most notably, we reveal that prevailing post-answer “double-checks” are largely superficial and rarely yield substantive revisions. A targeted intervention further shows that explicitly eliciting richer reflection processes can substantially improve failed self-correction. To support this large-scale study, we propose CAPO, an automated annotation method used to construct a dataset of 277,534 reasoning steps with strong agreement with human expert annotations. We further validate the main behavioural patterns on a newer reasoning model and a coding domain, demonstrating the broader applicability of the proposed taxonomy. All source code and data are available at https://github.com/hehepig4/psyche.

Multimodal clinical models have shown strong performance in dermatology recommendation and triage tasks, yet their robustness under demographic perturbations remains poorly understood. We introduce a clinician-validated counterfactual dermatology benchmark for evaluating recommendation stability under skin-tone perturbations in multimodal vision-language models. Our benchmark contains 240 physician-validated images from 60 clinical cases, each paired with four diagnosis- and severity-matched skin-tone variants constructed through iterative LLM filtering and physician review. Across diagnosis, management, urgency, and referral tasks, we observe substantial model-dependent instability even when clinicians verified preservation of diagnosis and severity across perturbation variants. Recommendation changes occur in up to 51.7% of paired comparisons, accompanied by shifts in uncertainty framing, pigmentation-sensitive terminology, and escalation-oriented language. Our findings suggest that demographic perturbations can influence not only diagnostic predictions, but also downstream recommendation behavior in multimodal clinical models. We release a controlled evaluation framework for studying recommendation robustness in medical AI systems.

To enhance the multi-turn defense capabilities of Large Language Models, recent studies focus on a “co-evolution framework.” This framework iteratively trains an attacker model to explore vulnerabilities and a target model to learn safety alignment. However, in our preliminary experiments, existing methods suffered from severe training instability. To address this and align with recent optimization methodologies, we introduce GRPO into the training loops of both models. We found that a single alignment cycle sufficiently established the target model’s defense, which in turn stalled the subsequent attacker training. Under the GRPO-based attacker-training pipeline and experimental setting examined in this study, this limitation rendered the iterative co-evolution process ineffective. Furthermore, to maintain model utility, we mix harmful goals with safe goals automatically rewritten from them. In our experiments, mixing safe goals improved utility but yielded higher attack success rates than training without them, revealing a safety-utility trade-off in our setting. Warning: This paper includes examples of harmful language.

While chain-of-thought prompting and scratchpads are frequently conceptualized as external memory for reasoning tasks, this perspective breaks down when a final answer must be generated from a bounded, jointly retained state. We formulate this limitation through a minimal constraint-satisfaction model in which an agent must exactly output N independent binary constraints using a shared internal and external memory configuration yielding at most 2^B distinguishable states. Under this framework, we mathematically prove a tight phase law:

p⋆(N,B)=min{1,2^B-N}.

This Joint Memory Budget Theorem establishes that each independent constraint exceeding the retained memory budget exponentially halves the probability of exact success. We further generalize this limit with adversarial robustness and a min-entropy extension that separates raw prompt length from effective independent information. To empirically test the qualitative predictions of this theory, we introduce a two-call retention protocol that forces large language models to reconstruct constraint hands solely from their bounded written representations. Comprehensive evaluations across modern long-context and reasoning models reveal sharp, capacity-dependent performance cliffs. While variables such as scratchpad formatting, effective task entropy, and answer-time external access can shift this phase boundary, they do not eliminate it. Ultimately, we demonstrate that long-context and scratchpad reasoning are fundamentally bottlenecked not by nominal context length, but by the independent information jointly available at the time of answer generation.

Feature engineering is central to tabular machine learning, but useful transformations must be both statistically predictive and semantically valid with respect to feature names, task descriptions, and observed data patterns. Although large language models can exploit such context, prompt-based feature engineering remains unreliable: generated transformations may be redundant, schema-invalid, non-executable, or weakly aligned with downstream performance. We propose MSDPO, a two-stage Direct Preference Optimization framework for automated feature engineering. Proposal-DPO aligns the model toward useful, diverse, and non-redundant feature proposals, while Function-DPO aligns it toward executable feature-construction code using execution and validation feedback. After alignment, the frozen model directly generates transformation functions for unseen tabular tasks. Across 20 classification datasets, MSDPO achieves the best overall mean rank among strong classical and LLM-based feature-engineering baselines. Under a budget-matched comparison inside the same feature-engineering harness, MSDPO also attains a better mean rank than Claude Sonnet 4.6, despite using an 8B open-weight backbone. Moreover, preferences constructed from classification signals transfer to four regression tasks. Ablation results show that the two DPO stages are complementary, improving both feature design and executable implementation.

Large language models (LLMs) have shown remarkable capabilities in interactive real-world software engineering (SWE) scenarios. However, the execution-aware training data are hard to scale and are mostly used for post-training, leaving agentic mid-training underexplored. To this end, we propose COMP-SYNTH, a novel automated pipeline for synthesizing reflective code completion trajectories. Specifically, based on executable repository environments, COMP-SYNTH identifies test-covered target functions, masks their implementations, and prompts an LLM to complete them under execution feedback. Then, each generated completion is verified by running the associated tests. Furthermore, failed attempts are augmented with diagnostic signals to extract useful supervision. Thus, we construct COMP-TRAJ, a mid-training dataset that exposes models to repository context, execution feedback, and iterative code revision. Extensive experiments on repository-level SWE benchmarks show that mid-training on COMP-TRAJ effectively enhances capabilities in SWE. Our code and data are publicly available at https://github.com/uuam5/comp-traj.

Lightweight neural network models deployed on edge devices are susceptible to performance degradation under unseen acoustic conditions due to domain mismatch between training and deployment environments. This is particularly relevant in causal low latency applications such as cochlear implants (CIs), where limited temporal context and strict computational constraints restrict model robustness. Domain adversarial learning can improve generalization by encouraging representations that are both task-discriminative and domain-invariant; however, jointly learning these representations becomes difficult in limited capacity models. We introduce an adapter-based adversarial learning framework for robust and parameter-efficient domain adaptation. The proposed approach incorporates an adapter module into a pretrained causal backbone. We first establish a robust pretrained baseline network and subsequently perform adversarial adaptation under mismatched acoustic conditions. We evaluate the proposed framework on framewise phonetic group classification to guide phoneme-based time-frequency mask estimation. Empirical results demonstrate consistent improvements in phoneme classification over both standard supervised training and conventional adversarial learning, particularly under low signal-to-noise ratio levels, resulting in statistically significant improvements in objective intelligibility measures of CI vocoded speech.

Generating molecules that satisfy precise numeric constraints over multiple physicochemical properties is critical and challenging. Although large language models (LLMs) are expressive, they struggle with precise multi-objective control and numeric reasoning without external structure and feedback. We introduce M^4olGen, a fragment-level, retrieval-augmented, two-stage framework for molecule generation under multi-property constraints. Stage I : Prototype generation: a multi-agent reasoner performs retrieval-anchored, fragment-level edits to produce a candidate near the feasible region. Stage II : RL-based fine-grained optimization: a fragment-level optimizer trained with Group Relative Policy Optimization (GRPO) applies one- or multi-hop refinements to explicitly minimize the property errors toward our target while regulating edit complexity and deviation from the prototype. A large, automatically curated dataset with reasoning chains of fragment edits and measured property deltas underpins both stages, enabling deterministic, reproducible supervision and controllable multi-hop reasoning. Unlike prior work, our framework better reasons about molecules by leveraging fragments and supports controllable refinement toward numeric targets. Experiments on generation under two sets of property constraints (QED, LogP, Molecular Weight and HOMO, LUMO) show consistent gains in validity and precise satisfaction of multi-property targets, outperforming strong LLMs and graph-based algorithms.

Search-based reasoning and chain-of-thought (CoT) prompting offer two contrasting inference-time paradigms for language models: CoT commits to a single linear trajectory, while search explores alternative partial reasoning paths before selecting an answer. In this work, we study this trade-off using A* as a controlled instantiation of heuristic-guided search. Across StrategyQA, LogicQA, HotpotQA, GSM8K, and MATH, and across three model scales, we find that search is not a universal upgrade over CoT; instead, the two strategies solve substantially different subsets of instances. To explain this complementarity, we introduce a Reasoning Topology Framework based on structural features computable before generation. A derived branching coefficient is strongly associated with when search helps across mixed-benchmark topology clusters, suggesting that the CoT–search boundary is partly explained by problem structure. We further train a topology-aware router that selects between CoT and A* without observing generated traces. Despite this restriction, it outperforms post-hoc routers based on semantic entropy and LLM-as-critic judgments. We attribute this to a Fluency–Correctness Asymmetry: fluent traces are often incorrect, making trace-level adjudication unreliable. Overall, our results suggest that adaptive reasoning should route by problem structure, deploying search only when the input topology warrants its higher cost. Code: https://github.com/newcodevelop/astar-cot-reasoning.

Large language models (LLMs) are widely used as automatic judges, with validity typically assessed via alignment with human scores. However, aggregate agreement fails to reveal whether humans and LLMs find the same evaluation cases difficult. In this paper, we study this problem in summarization evaluation from a psychometric perspective. We fit Many-Facet Rasch Models separately to human and LLM ratings to decompose scores into latent summary quality, rater severity, dimension severity, and rating-scale thresholds. Building on this decomposition, we define residual hardness as a model-adjusted measure of judging difficulty and compare whether human and LLM judges share the same hardness structure.

Across 17 open-weight LLM judges on SummEval, we find that moderate alignment in latent summary quality does not imply alignment in residual hardness. Human and LLM judges differ in which summary–dimension units remain difficult, and this mismatch is strongly dimension-dependent. Consistency shows a pronounced LLM-hard shift, whereas coherence shows a human-hard shift. We further show that human-easy but LLM-hard cases are partially predictable from observable source–summary properties. These findings suggest that aggregate human alignment reflects only part of LLM-as-judge reliability, while psychometric residual diagnostics support more informative judge evaluation and more targeted human–LLM collaboration.

Visual Question-Answering (VQA) is a challenging multimodal task that requires integrating visual and textual information to generate accurate responses. While multimodal Retrieval-Augmented Generation (mRAG) has shown promise in enhancing VQA systems by providing more evidence on both image and text sides, the default procedure that addresses VQA queries, especially the knowledge-intensive ones, often relies on multi-stage pipelines of mRAG with inherent dependencies. To mitigate the inefficiency limitations while maintaining VQA task performance, this paper proposes a method that trains a multimodal planning agent, dynamically decomposing the mRAG pipeline to solve the VQA task. Our method optimizes the trade-off between efficiency and effectiveness by training the agent to intelligently determine the necessity of each mRAG step. In our experiments, the agent can help reduce redundant computations, cutting search time by over 60% compared to existing methods and decreasing costly tool calls. Meanwhile, experiments demonstrate that our method outperforms all baselines, including a Deep Research agent and a carefully designed prompt-based method, on average over six various datasets. Code is released.

Activation steering is a lightweight alternative to fine-tuning for aligning and personalizing large language models (LLMs). Prior work such as Bi-directional Preference Optimization (BiPO) learns dense steering vectors from preference data. However, dense interventions can entangle latent factors due to neuron multi-semanticity, hurting stability and fine-grained control, thus limiting their effectiveness especially in cultural alignment, where closely related values and behaviors (e.g., across Middle Eastern cultures) must be distinguished. We propose Yet another Policy Optimization (YaPO), a reference-free method that learns preference-optimized steering vectors in the sparse latent space of a Sparse Autoencoder (SAE). While the learned vector is not explicitly constrained to be sparse, it acts over sparse SAE features, providing a more disentangled and interpretable intervention basis. By optimizing sparse codes, YaPO yields more disentangled and interpretable steering directions. Across cultural alignment and several alignment-related behaviors (e.g., hallucination reduction and jailbreak resistance), YaPO converges faster and is more stable than dense steering baselines while improving performance. Importantly, YaPO preserves general knowledge, with no measurable degradation on MMLU. Overall, our results show that YaPO provides a general recipe for efficient, stable, and fine-grained alignment of LLMs, with broad applications to controllability and domain adaptation.

Capabilities of large language models to generate multilingual coherent text have continuously enhanced in recent years, which opens concerns about their potential misuse. Previous research has shown that they can be misused for generation of personalized disinformation in multiple languages. It has also been observed that personalization negatively affects detectability of machine-generated texts; however, this has been studied in the English language only. In this work, we examine this phenomenon across 10 languages, while we focus not only on potential misuse of personalization capabilities, but also on potential benefits they offer. Overall, we cover 1080 combinations of various personalization aspects in the prompts, for which the texts are generated by 16 open-weights models drawn from 5 architecturally distinct families, including multiple size variants per family to assess scale effects (17,280 texts in total). Our results indicate that there are statistically significant differences in personalization quality of the generated texts when targeting demographic groups and when targeting social-media platforms across languages. Personalization towards platforms affects detectability of the generated texts in a higher scale, especially in English, where the personalization quality is the highest.

While large vision-language models excel at optical character recognition, it remains unclear how their underlying vision encoders process text and its meaning, and whether they support anything resembling reading comprehension. We systematically study transformer-based vision encoders, using cross-modal alignment between rendered text and visual concepts together with probing and retrieval-based analyses as a controlled window into these broader capabilities. Our layer-wise analyses span orthographic, grammatical, and semantic organization, and reveal that strong cross-modal mapping emerges only in the deepest layers, after unimodal object and text recognition have already developed, coinciding with a progression from surface-level to semantic representations. Although this hierarchy is strongest in contrastively trained models, larger self-supervised models and text-rich training data partially support similar behaviors without explicit language supervision. Together, our results suggest that increasing model and training-data scale, as well as text-rich training data, can partially compensate for the absence of language supervision.

Solid Earth geoscience, geology, geochemistry, and geophysics, is increasingly served by large language models (LLMs) and multimodal LLMs (MLLMs) that combine text, geological maps, hyperspectral imagery, 3D structural models, spatial databases, and quantitative measurements into operational decision pipelines. Existing studies are fragmented across tasks, datasets, and system designs. This survey provides a structured review of LLM/MLLM research in solid-Earth geoscience along three axes: applications, modalities, and Memory-Reason-Act techniques. We identify five clusters of application (information extraction, knowledge Q A and retrieval, multimodal map and 3D interpretation, spatial predictive modelling, and agentic pipelines), characterise four modalities (text, visual, spatial, and quantitative signal) and their encoding gaps, and analyse Memory, Reasoning, and Action design choices with their failure modes. We further examine data resources and bias, and identify the absence of closed-loop drill-outcome validation as the central methodological gap blocking next-generation solid-Earth foundation models.

Reasoning-oriented language models can generate thousands of tokens before answering, yet the internal factors that determine their reasoning length remain poorly understood. This paper studies whether reasoning length can be controlled through sparse neuron interventions, complementing prompt-based, budget-based, and controller-based approaches. The proposed method discovers reasoning-length-associated neurons from paired long and concise correct trajectories on a training split, ranks MLP neurons by long-minus-concise activation contrast, and tests the selected neurons on disjoint evaluation splits by scaling their activations during generation. The evaluation covers math, science, and commonsense reasoning tasks, including GSM8K, MATH500, ARC-Challenge, CommonsenseQA, and QASC, under a strict split-separated protocol that never discovers neurons on held-out evaluation examples. In several, but not all, task/model/budget regimes, selected neurons modulate generated length differently from layer-matched random controls, with a modest single-run specificity result on CommonsenseQA accompanied by a small accuracy increase. These results provide task-, model-, and prompt-protocol-specific evidence that sparse MLP neurons can, in some split-controlled regimes, provide neuron-level control signals for generated reasoning length.

While dictionary-based approaches have shown promise in applying Large Language Models (LLMs) to generation tasks such as Neural Machine Translation (NMT), existing studies overlook several key challenges. Most methods rely on full or rule-selected dictionaries for prompting, assume that all dictionary entries are effectively utilised by LLMs, and construct dictionaries from bilingual parallel corpora, each of which limits performance. We propose a new dictionary-based translation framework that addresses these issues by fine-tuning LLMs with LoRA to autonomously select salient dictionary entries, choosing translations with the highest dictionary utilisation, and introducing a Multilingual Lexicon Induced Dictionary that leverages multilingual parallel corpora to improve dictionary quality. Experiments on the FLORES benchmark across multiple languages demonstrate the superior performance of our framework.

Multi-document diversity summarization (MDDS) in high-stakes domains such as crisis response demands fundamentally different capabilities than consensus-based summarization. While standard summarization focus on fluency and faithfulness, high-stakes tasks would also require: maximizing coverage of divergent information, identifying unique information clusters, and providing verifiable citations to source documents. Existing evaluation frameworks fail to capture these task-specific requirements. Our study introduces a comprehensive evaluation framework for high-stakes MDDS, validated on a multi-source crisis dataset that we developed covering ten crisis events. We propose novel metrics, Comprehensive Coverage Score, Frequency-Salience Correlation, and Citation-Frequency Correlation, to diagnose critical failure modes in six off-the-shelf large language models (LLMs). Our results allow us to identify three critical failure modes of LLMs: (1) a consensus trap in coverage: where models capture high-level information clusters, but miss specific, actionable details needed for decision-making; (2) majority bias, where more frequent information are prioritized over rare but important ones and; (3) a verifiability gap, where citations of rare claims are less reliable

Modern enterprises store vast, heterogeneous data across large data lakes, where the central challenge is shifting from answering an isolated question to discovering and assembling the tables and documents that together serve an analytical goal. We refer to such a coherent collection of related assets as a data product. Existing table-text benchmarks evaluate factoid question answering over individual tables and therefore do not capture this discovery and assembly problem. We introduce DPDisc, the first benchmark for data product discovery: given a high-level Data Product Request (DPR), a system must retrieve the coherent set of tables and passages that satisfy it. DPDisc is constructed by DPForge, an automated pipeline that repurposes table-text QA datasets by clustering related tables and passages into products, generating DPRs with an LLM ensemble, and validating them through multi-phase LLM evaluation. It comprises 13,076 validated instances with full provenance, spanning open-domain and financial sources. Sparse, dense, and hybrid baselines establish that the task is feasible yet reveal large performance gaps across domains, motivating structure-aware retrieval.

Relying on unverified, third-party data for training large language models or outsourcing fine-tuning to untrusted parties poses a severe risk to the model’s integrity, giving rise to so-called data-poisoning attacks that introduce backdoors in LLMs. In this paper, we investigate to which extent fine-tuning can be hardened by using the base model’s intrinsic generative capabilities. We sketch a simple defense strategy using mere prompt engineering that—in contrast to prior work—neither requires knowledge of the exact backdoor trig- ger used nor clean data as a reference. Instead, we leverage the assumption that (a) potential attackers aim for unsafe outputs and (b) triggers induce language incoherence. Across downstream tasks and attack types, we show that such a simple strategy can mitigate backdoors more effectively than existing defenses. In comparison to post-hoc defenses, our method substantially reduces the attack success rate up to 85% with Llama-3 while consistently preserving a high natural accuracy.