Yaoyang Ye

Author directory

2026

Large language models (LLMs) are increasingly applied to vulnerability detection in real-world software systems, where the task goes beyond binary classification and requires calling context exploration over large code repositories. However, existing benchmarks primarily formulate the vulnerability detection as a classification problem without calling contexts and provide no formalized explanations, making them inadequate for evaluating LLM-driven vulnerability detectors upon repositories. To fill the gap, we introduce ManBo, a benchmark designed to assess the capability of LLM-driven vulnerability detectors in the real-world scenarios. Specifically, ManBo provides human-verified vulnerability explanations together with calling contexts, enabling principled evaluation of the ability to both identify vulnerabilities and explain root causes. It contains 205 vulnerable instances and 164 safe instances collected from 74 high-profile C/C++ projects, covering three critical types of memory corruption. We evaluate a broad spectrum of LLM-driven detectors on ManBo, and find that effective calling context retrieval is essential for repository-level vulnerability detection, exposing a fundamental gap between the capabilities of existing LLM-driven detectors and real-world requirements. ManBo is publicly available at https://github.com/jinyaoguo/ManBo.