Frontier LLMs Struggle in Cybersecurity Benchmarks

A recent study from arXiv examines the readiness of advanced large language models for cybersecurity applications. The researchers created a dual-mode benchmark that includes white-box function-level vulnerability detection in C, Java, and Python (VulnLLM-R) and black-box web application security testing on five production-like applications, identifying 118 actual vulnerabilities across more than 20 CWE families. They assessed six leading models—GPT-5.4, Codex~5.3, Claude Opus~4.6, Sonnet~4.6, Gemini~3.1~Pro, and Gemini~3~Flash—along with two specialized models across four testing methods. Findings reveal that all frontier models exhibit false positive rates between 10-50% in white-box detection and only achieve 4-8% ground-truth coverage in black-box testing, which slightly increases to 10-19% with external tools. The study suggests that employing structured penetration-testing methodologies in specialized agents may enhance results, but overall, current frontier LLMs are not adequately prepared for practical cybersecurity tasks.