LLM-Authored Test Suites Found Deficient in New arXiv Study

A recent study published on arXiv (2608.01000) examines how well language models can create test suites, answer keys, rubrics, and reward functions, which are essential for determining correctness in other systems. The findings indicate that when following the standard deployment method (one-shot greedy authoring without reasoning during testing), models struggle significantly more with generating acceptable sets compared to assessing whether a candidate fits within a set. This discrepancy is consistent across four reference constructions—two featuring complete finite truth, one with a robust executable reference (HumanEval+/MBPP+), and one with an explicitly incomplete lexical reference (WordNet). The gap in performance ranges from +0.34 to +0.29 F1 across a 24x parameter spectrum and remains persistent. In the context of executable code, models that evaluate at F1 scores of 0.74-0.90 produce suites that only include 19-42% of oracle-correct solutions. A control experiment reveals that when instructed to generate the predicate instead of the set, performance improves, indicating that the challenge lies in set enumeration rather than predicate comprehension. This research underscores a significant limitation in employing LLMs as evaluators, raising concerns for AI assessment and safety.