Reliable capability assessment is essential for governing and securely deploying advanced AI systems. However, "AI Sandbagging," where models strategically suppress their true capabilities during evaluation, undermines existing assessment frameworks by making it difficult to distinguish genuine limitations from strategic concealment. To address this challenge, we propose a new black-box sandbag detection method inspired by "honesty inertia," a psychological phenomenon where humans struggle to maintain deception after answering simple questions truthfully. We hypothesize that large language models exhibit an analogous effect: inserting several simple, unrelated questions, which target models should answer correctly, before the target query induces honest inertia, causing sandbag models to involuntarily increase accuracy while standard models remain stable. Through extensive experiments on multiple datasets with various model architectures, we confirm the validity of this hypothesis and establish this differential response as a reliable, model-independent sandbag detection signal. We further provide interpretability support for this phenomenon from two dimensions: theoretical derivation and hidden state analysis of models. This work offers a practical tool for enhancing AI audit and supervision.

Although recent large reasoning models (LRMs) achieve strong performance, their reasoning processes remain vulnerable to external manipulation. We observe that LRMs can gradually rationalize incorrect user-provided hints during multi-step reasoning. Instead of rejecting false hints, models adapt their intermediate states to accommodate them and eventually produce seemingly coherent yet incorrect conclusions. We refer to this phenomenon as self-brainwashing and analyze it through empirical and theoretical analyses. We find that unfaithful reasoning steps are often locally detectable when models only focus on local reasoning context. However, this ability degrades over long reasoning trajectories, allowing unfaithful steps to accumulate and bias the final answer toward the hint. Motivated by these findings, we propose Long-Short Memory Steering (LSMS), a training-free inference-time framework that uses short-term reasoning hidden states to enhance faithfulness and long-term reasoning hidden states to maintain consistency. Across three open-source reasoning models and four reasoning datasets, LSMS consistently reduces unfaithful answer shifts while preserving benign accuracy. On average, it lowers the shifted ratio by more than 20 percentage points under three real-world hint-based threats.

In-the-real-world datasets provide a crucial reflection of real-world adversarial distributions and hold significant practical importance for evaluating the safety of T2I models. However, existing in-the-real-world unsafe benchmarks are not tailored for T2I generation: many prompts lack visual specificity or concrete scenarios, producing distorted and low-fidelity images that are misclassified as safe by automated detectors. This acts as a confounder of successful safety alignment and compromises the accuracy of safety measurement. To address this, we propose RW2Unsafe, an automated three-module framework that reformulates source in-the-real-world posts into visually descriptive T2I prompts while preserving the original harmful intent. We use it to construct rw_t2i_4chan, a benchmark of 2,000 high-quality in-the-real-world unsafe T2I prompts. Auditing five advanced T2I models, we find that prior approaches relying on source prompts systematically underestimate safety risks: on Bagel, for instance, the mean unsafe score under source prompts is under one-third of that under rewritten prompts. We hope the proposed framework and dataset could contribute to more accurate safety measurement and safer T2I models.

As high-quality data becomes harder to obtain, reward models are increasingly important. Beyond the costly RLHF stage, they are now used at inference time to guide LLM generation and in data selection for post-training. These methods bring efficiency and performance gains, but current reward models often fail to prevent untrustworthy behaviors such as privacy leaks and stereotypes. Re-training reward models to address these issues is expensive, since it requires large-scale human preference data. We propose SelfRW, a lightweight intrinsic reward that needs no extra fine-tuning or auxiliary models. By pruning current LLMs to approximate an “trust” and an “untrust” token distribution, we compute the log-probability difference as an auxiliary reward. When integrated into reward-guided sampling, SelfRW significantly reduces untrustworthy outputs while preserving task performance. It also improves reward-guided data selection, yielding better post-trained models. Experiments with two reward models and four LLMs on privacy, bias, and stereotype benchmarks show that combining SelfRW consistently improves trustworthiness (over 10% in privacy tasks and 20% in bias tasks) with minimal impact on general utility benchmarks.

Diffusion Language Models (DLMs) benefit from a flexible decoding mechanism that enables parallelized inference and controllable generations over autoregressive models. Yet such flexibility introduces a critical challenge: inference performance becomes highly sensitive to the decoding order of tokens. Existing heuristic methods, however, focus mainly on local effects while overlooking long-term impacts. To address this limitation, we propose the Foreseeing Decoding Method (FDM), a novel approach that integrates both local and global considerations to unlock the full potential, employing a search-based strategy to enable effective optimization in discrete spaces. Furthermore, by analyzing the consistency of chosen tokens in the full decoding process, we develop a variant, FDM with Acceleration (FDM-A), which restricts deep exploration to critical steps identified as the exploration and balance circumstances. Extensive experiments across diverse benchmarks and model architectures validate the scalability of FDM and demonstrate the superior efficiency-performance trade-off achieved by FDM-A. Our work might potentially provide a principled step toward more powerful decoding methods for DLMs. Our code is available at https://github.com/PKU-ML/FDM.

Diffusion Language Models (DLMs) are emerging as competitive alternatives to auto-regressive models, but their flexible, any-order decoding may introduce new trustworthiness risks. We present TrustDLM, a comprehensive benchmark for evaluating DLM trustworthiness across safety, privacy, and fairness with multiple categories of static post contexts. Our results show that while DLMs are generally trustworthy with user prompts alone, their alignment degrades noticeably when malicious post-contexts are attached to masked responses. We further find that decoding order and generation length affect evaluation outcomes, while longer contexts do not necessarily induce stronger effects. Finally, we introduce TrustDLM-Auto, the first automatic framework for identifying vulnerable DLM configurations, which reveals substantial weaknesses across models and dimensions. Overall, TrustDLM provides the first trustworthiness evaluation framework specifically designed for DLMs, with practical applicability further validated on closed-source DLMs. Our work will help the community enhance the trustworthiness of DLMs. Our code is available at https://github.com/PKU-ML/TrustDLM.

Large language models increasingly rely on long chain-of-thought (CoT) trajectories for complex reasoning, but autoregressive generation brings substantial memory and inference costs. Latent reasoning models (LRMs) offer a more efficient alternative by compressing intermediate reasoning into a small number of continuous latent vectors. Despite their efficiency, however, the adversarial robustness of LRMs remains largely underexplored. In this work, we systematically evaluate the robustness of latent reasoning across textual and multimodal settings, covering eight models and six benchmarks. We find that, across our evaluated settings, LRMs are generally less robust than explicit CoT baselines under adversarial perturbations, with particularly severe degradation under white-box attacks. Further analysis reveals distinct failure modes across modalities: textual latent states exhibit brittle dynamics and high sensitivity to specific input patterns, while latent states in multimodal models can remain largely invariant to input perturbations and have limited influence on final predictions. These findings expose robustness limitations of current latent reasoning approaches and highlight the need to jointly consider efficiency and robustness when designing implicit reasoning systems. We have open-sourced our code to facilitate reproduction of our research https://github.com/PKU-ML/latent-reasoning-model-assessment.

Accurately assessing jailbreak success remains unsolved, as existing automatic methods often diverge from human perception. To address this, we propose JADES, a universal evaluation framework that decomposes a harmful query into weighted sub-questions, scores each sub-answer, and aggregates the scores into a final decision; an optional fact-checking module further mitigates hallucination-induced misjudgments. We validate JADES on JailbreakQR, a meticulously annotated benchmark of 400 prompt–response pairs. Under binary settings (success/failure), JADES achieves 98.5% human agreement, outperforming strong baselines by over 9%. Re-evaluating five popular attacks across seven LLMs reveals systematic overestimation of jailbreak. JADES also enables granular ternary evaluation (failure/partial success/success), uncovers that partial successes dominate current reported successes, and recalibrates the perceived severity of existing attacks. Overall, JADES reveals a major limitation of current jailbreak evaluation and, together with ternary assessment, enables a more accurate understanding of the true jailbreak effectiveness.

Machine learning models are increasingly deployed in real-world applications, but even aligned models such as Mistral and LLaVA still exhibit unsafe behaviors inherited from pre-training. Current alignment methods like SFT and RLHF primarily encourage models to generate preferred responses, but do not explicitly remove the unsafe subnetworks that trigger harmful outputs. In this work, we introduce a resource-efficient pruning framework that directly identifies and removes parameters associated with unsafe behaviors while preserving model utility. Our method employs a gradient-free attribution mechanism, requiring only modest GPU resources, and generalizes across architectures and quantized variants. Empirical evaluations on ML models show substantial reductions in unsafe generations and improved robustness against jailbreak attacks, with minimal utility loss. From the perspective of the Lottery Ticket Hypothesis, our results suggest that ML models contain “unsafe tickets” responsible for harmful behaviors, and pruning reveals “safety tickets” that maintain performance while aligning outputs. This provides a lightweight, post-hoc alignment strategy suitable for deployment in resource-constrained settings.