BLAST: New Benchmark Evaluates LLMs on ASP Code Generation

Researchers have introduced BLAST, the first dedicated benchmarking methodology and dataset for evaluating the accuracy of large language models (LLMs) in generating Answer Set Programming (ASP) code. The framework features two novel semantic metrics tailored to ASP code generation. An empirical evaluation involved ten well-established graph-related problems from the ASP literature and eight state-of-the-art LLMs. The work addresses a gap in assessing LLM performance on declarative programming paradigms.