Open-Source Framework Evaluates LLM-Generated Cybersecurity Detection Rules

An innovative open-source framework for evaluation and benchmark metrics has been launched to measure the performance of Large Language Models (LLMs) in creating cybersecurity detection rules. This framework, outlined in a paper on arXiv (ID 2509.16749), utilizes a holdout set-based approach to juxtapose LLM-generated rules with those created by humans. It introduces three essential metrics influenced by expert evaluation techniques, delivering a comprehensive assessment of LLM-driven security rule generators. The methodology features examples from Sublime Security's detection team and its Automated Detection Engineer (ADE), with an in-depth evaluation of ADE's capabilities included in the findings. This initiative responds to the increasing use of LLMs in security, where inadequate effectiveness measures limit trust among practitioners. The framework aspires to bolster confidence in AI security tools through standardized evaluation processes. The paper falls under Computer Science > Cryptography and Security and contains submission history, references, and citation tools, emphasizing the necessity of thorough evaluation in the integration of AI in vital security functions.