Porting Benchmark Study Reveals Performance Gaps in Automated Security Patch Backporting Tools

A new benchmark called Porting Benchmark is introduced in an arXiv paper, offering a curated dataset of 1,234 security patch backporting cases. These cases span cross-version, cross-branch, and cross-repository scenarios, and are paired with a common evaluation framework. The study evaluates five tools under aligned settings, covering program analysis, LLM prompting, and LLM agents. The results demonstrate that evaluation conditions significantly affect reported performance. While earlier tools claimed success rates above 80% on their own datasets, the aligned evaluation shifts the apparent performance landscape. Specifically, PortGPT and TSBPort maintain comparatively strong performance on the Replication Dataset, whereas FixMorph and Mystique degrade substantially under the coordinated evaluation. This work addresses the issue of evaluations being confined to homogeneous environments, such as a single repository or specific project versions, which leaves generalization unclear. By providing a standardized benchmark, Porting Benchmark enables more reliable comparisons and highlights the importance of robust evaluation methodologies for security patch backporting. The paper is identified as arXiv:2608.17671 and is available on the arXiv platform.