Open-Weight LLMs Outperform Symbolic Baselines in PDDL Planning Model Repair
A study available on arXiv (identifier 2608.17341) investigates the use of open-weight large language models (LLMs) to fix errors in Planning Domain Definition Language (PDDL) models, which are essential for AI planning. It focuses on detecting and correcting mistakes through both positive and negative test plans. The research compares an LLM-only method to a symbolic baseline, which achieves an F1 score of 0.49. The top-performing LLM, requiring significant reasoning effort, reaches a score of 0.87, reflecting a 0.38 enhancement, although its average test pass rate is 0.82, plummeting to 0.06 in the Thoughtful domain. Results indicate that while LLMs can excel over symbolic techniques in certain cases, their reliability varies by domain, highlighting the necessity for additional investigation.
Key facts
- Paper available on arXiv with identifier arXiv:2608.17341
- Study evaluates open-weight large language models for PDDL model repair
- Method uses an LLM-only approach
- Symbolic baseline achieves F1 score of 0.49
- Best-performing LLM with high reasoning effort achieves F1 score of 0.87
- Absolute improvement in F1 score is 0.38
- Mean test pass rate for the best setting is 0.82
- On the Thoughtful domain, test pass rate falls to 0.06
Entities
—