Existing code vulnerability detection (CVD) methods face severe challenges in terms of generalization. Most of these methods directly map the source code to a binary vulnerability label or a vulnerability explanation, struggling to capture the deep features within the complex relationship between them. Thus, these works fail to generalize to out-of-distribution data well. To address this issue, we propose a simple yet effective training framework called refinement-integrated reasoning (RIRE), leveraging refinements that indicate the patched code in the CVD task. Firstly, RIRE introduces the refinement as a guiding signal to help LLMs synthesize reasoning chains as explanations, which align with the vulnerability label, and uses these data for supervised fine-tuning. Secondly, refinement is utilized through a hierarchical preference optimization framework to help determine which reasoning-enhanced detection result is of higher quality, further mining essential features for fine-grained distinguishability and detection capability. Experimental results across five benchmark datasets demonstrate that our proposed RIRE framework achieves superior performance over strong baselines, especially being more generalizable to unseen datasets during training.

Dynamic benchmarking is an essential research direction in the evaluation of large language models (LLMs) to avoid data contamination. Although existing works have proposed LLM-based methods for dynamic benchmarking, there still lacks a unified paradigm for performance comparison among different LLMs, which hinders researchers from model selection for specific domains and tasks. To address this challenge, we propose a comprehensive evaluation suite called DyBenchEval, including a novel seed dataset across 4 domains and 7 tasks as well as multi-aspect evaluation metrics. A cross data filling method is utilized for high-quality data augmentation with domain/task-level similarity, aiming to overcome the data scarcity problem. For automatic metrics, we consider coherence, correctness, and diversity of LLM-generated evaluation samples, simultaneously measuring the respective quality of generated questions / answers and the relationship between them. Extensive experimental results on DyBenchEval reveal deficiencies in 11 mainstream closed-source and open-source LLMs when dealing with dynamic benchmarking across different domains and tasks.