WeSCE: Benchmarking Security Drift in LLM-Driven Code Editing

A new benchmark named WeSCE has been developed by researchers to assess security drift in code modifications made by large language models (LLMs) operating under weak-security conditions, where only functional goals are outlined without specific security criteria. This benchmark includes 400 executable programs sourced from actual code, focusing on tasks like feature addition, feature removal, bug fixing, and refactoring. To evaluate security drift, the researchers suggest a continuous risk representation that integrates various vulnerability signals into a single framework. They establish drift metrics that reflect shifts in overall risk, extreme severity, and vulnerability distribution amid code alterations, offering insights into security from average to worst-case scenarios. This initiative aims to tackle the rising issue of potential security vulnerabilities introduced by LLM-assisted code editing. Developers and researchers can utilize this benchmark to evaluate and reduce such risks. The paper can be found on arXiv in the Computer Science > Cryptography and Security section.