VICBench: New Multi-Language Benchmark for Code Vulnerability Detection
VICBench, a novel benchmark dataset, has been unveiled by researchers to assess tools for detecting security vulnerabilities. This dataset includes 100 verified vulnerability-inducing commits (VICs) linked to 100 CVEs across 88 projects written in Python, Java, and C++, encompassing 48 types of CWEs. Developed through a combination of expert human annotation and an agentic workflow, VICBench overcomes the shortcomings of previous datasets, which often suffer from limited programming language representation, simplistic patch complexity, and narrow project diversity. The benchmark showcases intricate real-world vulnerability fixes, averaging 38.6 lines, with corresponding VICs averaging 252.5 lines, significantly larger than earlier datasets. Evaluation of advanced algorithms V-SZZ and LLM4SZZ on VICBench yielded F1 scores between 33.3% and 40.1%, highlighting the ongoing need for considerable manual intervention. This benchmark seeks to enhance the evaluation of vulnerability detection tools by offering a more thorough and realistic dataset. The research paper can be accessed on arXiv with the identifier 2608.12246.
Key facts
- VICBench contains 100 verified vulnerability-inducing commits (VICs) for 100 CVEs.
- The dataset covers 88 projects in Python, Java, and C++.
- It includes 48 CWE types.
- VICBench was created through dual annotation by human experts and an agentic workflow.
- The average fix size is 38.6 lines, and the average VIC size is 252.5 lines.
- State-of-the-art algorithms V-SZZ and LLM4SZZ achieve only 33.3%-40.1% F1 on VICBench.
- Existing vulnerability datasets suffer from limited language coverage, restricted patch complexity, and narrow project scope.
- The paper is available on arXiv with identifier 2608.12246.
Entities
Institutions
- arXiv