SWE-Mutation Benchmark Evaluates LLM-Generated Test Suites
A new benchmark called SWE-Mutation has been introduced to evaluate the quality of test suites generated by large language models (LLMs) for software engineering tasks. The benchmark uses systematically mutated solutions designed to "fool" test suites and pass validation, addressing the bottleneck of scarce high-quality test suites. LLM-generated test suites are often superficial and lack discriminative power, hindering program repair and reinforcement learning. SWE-Mutation aims to be a first step toward constructing high-quality test suites.
Key facts
- SWE-Mutation is a benchmark for evaluating LLM-generated test suites.
- It uses systematically mutated solutions to test discriminative power.
- High-quality test suites are scarce due to high annotation cost.
- LLM-generated test suites tend to be superficial.
- Test suites are needed for program repair and reinforcement learning.
- The benchmark addresses a key bottleneck in scaling LLM capabilities.
- The paper is available on arXiv as 2605.22175.
- The announcement type is cross.
Entities
Institutions
- arXiv