DeepRed Benchmark Evaluates LLM Agents in Cybersecurity Capture The Flag Challenges

A new open-source benchmark called DeepRed has been developed to assess the capabilities of Large Language Model agents in realistic cybersecurity scenarios. This evaluation framework places LLM-based agents in isolated virtualized environments running Kali Linux attacker tools, connecting them to target challenges through private networks. To provide more nuanced assessment than simple solved/unsolved metrics, researchers introduced a partial-credit scoring system based on challenge-specific checkpoints derived from public writeups. The benchmark includes an automated summarise-then-judge labeling pipeline that analyzes execution logs to determine checkpoint completion. Ten commercially accessible LLMs were tested using DeepRed across ten VM-based Capture The Flag challenges spanning multiple categories. The research addresses the growing interest in autonomous cybersecurity applications while acknowledging that current understanding of LLM agent capabilities in offensive settings remains limited. Full execution traces are recorded during testing for detailed analysis. The benchmark allows optional web search capabilities alongside terminal tools. The study was announced as arXiv:2604.19354v1, representing new research in the field of AI and cybersecurity.