Reasoner-Guided Repair Outperforms Corrective Hints for LLM Overcaution on OWL~2~DL Negations
A new study on arXiv (2604.23398) highlights a recurring mistake in GPT-5.4 related to OWL~2~DL compliance queries. The model frequently replies with "unknown" when the accurate answer, according to the reasoner, is "no," especially in situations involving FunctionalProperty closure or class disjointness. Researchers examined 180 queries validated by a reasoner and 18 queries created from insurance and clinical contexts, looking at four interaction strategies within a matched query budget: single-shot, three generic retries, three rounds with a reasoner hint, and three rounds without the hint. Findings showed direct accuracy at 43.9%, generic retries at 81.7%, hint strategy at 67.2%, and the verdict-only approach at 97.8%. All comparisons were significant, suggesting that hints might lower performance, while repairs without hints are very effective.
Key facts
- GPT-5.4 frequently answers 'unknown' on OWL~2~DL compliance queries where reasoner-entailed answer is 'no'.
- Error pattern occurs under FunctionalProperty closure or class disjointness.
- 180 reasoner-audited queries from procedural expansion plus 18 hand-authored held-out queries in insurance and clinical domains.
- Four interaction modes compared: single-shot, generic retry, verdict repair with OWA hint, verdict repair without hint.
- Direct faithfulness: 43.9% (CI 36.8-51.2).
- Generic retry faithfulness: 81.7% (CI 75.4-86.6).
- Verdict-with-hint faithfulness: 67.2% (CI 60.1-73.7).
- Verdict-only faithfulness: 97.8% (CI 94.4-99.1).
- All pairwise comparisons significant.
- Corrective hints can worsen LLM performance on OWL~2~DL queries.
Entities
Institutions
- arXiv