Active Testing for LLMs via Approximate Neyman Allocation

A new active testing algorithm for generative tasks in large language models (LLMs) has been introduced. The method uses semantic entropy from surrogate models to stratify the evaluation pool, then applies approximate Neyman allocation based on surrogate signals. This approach aims to reduce evaluation costs by selecting a small but informative subset of data. Tests across multiple language and multimodal benchmarks show significant improvements over existing methods, which primarily target classification tasks and fail on generative ones. The work addresses the growing need for efficient LLM evaluation as model scales and expert annotation costs rise.