New Metric Measures Buggy Code's Misguidance Effect on LLM Unit Tests
A new study from arXiv (2607.22883) introduces a metric to quantify the 'misguidance effect,' where buggy code prompts cause LLMs to generate unit tests that validate erroneous behavior rather than expose it. The analysis shows a twofold impact: increased misguided tests asserting incorrect behavior and suppressed generation of effective bug-finding tests. The effect is corroborated from a model-internal perspective, showing buggy code skews LLM preferences toward tests that assert the same errors. The paper proposes a specification-based unit test generation paradigm to counter this.
Key facts
- arXiv paper 2607.22883 introduces a metric for the misguidance effect
- Buggy code prompts cause LLMs to generate tests that validate errors
- Twofold impact: more misguided tests, fewer bug-finding tests
- Model-internal analysis confirms preference skew toward erroneous assertions
- Specification-based paradigm proposed as mitigation
Entities
Institutions
- arXiv