Xiangbo Zhang

Author directory

2026

Evaluating personality-related tendencies in Large Language Models (LLMs) helps characterize model behavior, compare models beyond task accuracy, and support responsible deployment in socially interactive settings. However, existing questionnaire-based evaluation methods exhibit limited stability and offer little explainability, as their results are highly sensitive to minor variations in prompt phrasing or role-play configurations. To address these limitations, we propose an internal-activation–based approach, termed Persona-Vector Neutrality Interpolation (PVNI), for stable and explainable personality trait evaluation in LLMs. PVNI extracts a persona vector associated with a target personality trait from the model’s internal activations using contrastive prompts. It then estimates the corresponding neutral score by interpolating along the persona vector as an anchor axis, enabling an interpretable comparison between the neutral prompt representation and the persona direction. We provide a theoretical analysis of the effectiveness and generalization properties of PVNI. Extensive experiments across diverse LLMs demonstrate that PVNI yields substantially more stable personality trait evaluations than existing methods, even under questionnaire and role-play variants.
Memorization in language models is widely studied but remains difficult to isolate and control. Understanding when and what models memorize is essential for explaining their predictions, yet existing approaches are post-hoc: they can detect memorization in trained models, but cannot disentangle its effects from architecture, data, or optimization. We introduce Memory Dial, a training framework that makes memorization an explicit, controllable variable. Memory Dial interpolates between standard cross-entropy and a temperature-sharpened objective via a single parameter, producing a family of models identical in architecture, data, and optimization, but varying in memorization pressure. Experiments across six architectures and five benchmarks demonstrate that: (1) reliably controls memorization, with seen-example accuracy increasing monotonically while unseen accuracy remains stable; (2) larger models are more responsive to memorization pressure; and (3) frequent sequences are easier to memorize than rare ones. Memory Dial provides a controlled experimental framework for studying how memorization behavior emerges and interacts with generalization in language models.
Retrieval-augmented medical vision-language models are increasingly used to support clinical reasoning when patient information is incomplete. However, retrieved cases are observations from other patients, not direct evidence for the target patient. In this paper, we study a source-attribution failure that we call retrieval-induced mirage: under missing target modalities, a model may copy findings from retrieved complete cases and present them as if they were observed in the target patient. We introduce RIM-BENCH, a counterfactual evaluation benchmark that fixes a controlled missing-modality setting while varying retrieval consistency and source-awareness. Unlike standard diagnostic accuracy evaluations, RIM-BENCH measures whether generated clinical claims are attributed to the correct evidence source. We define metrics for unavailable-modality claims, retrieved-to-target misattribution, evidence-source accuracy, retrieval-following under conflicting evidence, and missing-modality awareness. Across 1,200 target instances and three main models, implicit RAG raises unavailable-modality claims from 20.0% to 36.3% and induces a 26.2% RIMR, while source-attribution prompting lowers RIMR to 11.4% but does not eliminate the failure. Our framework highlights a safety risk for retrieval-augmented multimodal diagnosis: retrieval can be helpful as statistical context, but unsafe when models verbalize retrieved evidence as patient-specific observations.
Long-horizon language agents increasingly report intermediate progress while searching, editing, executing tools, and producing artifacts. Users rely on these reports to decide whether to intervene, continue, or accept an output, yet final task success does not reveal whether the agent described its work faithfully along the way. We study Progress Drift, a failure mode in which reported task state diverges from observable evidence. We introduce ProgressBench, a claim-level benchmark that aligns progress reports with milestones, tool observations, and artifacts across coding, web research, and research-replication tasks. We further propose Progress Ledger, an evidence-linked reporting protocol in which material progress claims carry evidence pointers, timestamps, verification status, and unresolved risks. In the reported evaluation across 1,280 tasks, ledger-constrained reporting is associated with higher Progress Claim Accuracy (80.6% to 89.7%) and lower False Completion Rate (12.7% to 6.9%) under shared tool budgets. These descriptive results suggest that agent evaluation should measure not only task completion, but also whether agents faithfully report what has and has not been achieved during execution.