Program Distillation Replaces LLM-as-a-Judge for Scalable Evaluation

A novel technique known as program distillation presents a more cost-effective, quicker, and clearer option compared to using LLM-as-a-judge for automated assessments. Rather than relying on a large language model for each query, this method extracts its decision-making process into a group of programs that evaluate candidates directly. The creators have introduced PAJAMA, a system that generates these programmatic judges, consolidates their evaluations, and incorporates a fallback system to refer low-confidence cases to an LLM. Evaluations conducted across five datasets and four model families indicate that these programmatic judges perform on par with a 13B-size LLM judge. The findings are available on arXiv (2607.22561).