NL2SHACL-Bench: New Benchmark Suite for Natural Language to SHACL Translation
A new benchmark suite named NL2SHACL-Bench has been developed by researchers to assess how effectively natural language requirements can be translated into SHACL, which is essential for validating RDF knowledge graphs. This suite fills a gap in the availability of specific benchmarks for this task and tackles the difficulty of evaluating generated shapes beyond mere string comparisons, as semantically similar shapes may vary in serialization and structure. The team tested four advanced large language models (LLMs) using NL2SHACL-Bench. Findings reveal that while these LLMs can generate syntactically correct SHACL, they often fail to create semantically equivalent constraints for intricate logical and structural patterns. The research paper can be found on arXiv with the identifier 2608.07530.
Key facts
- NL2SHACL-Bench is a benchmark suite for natural language to SHACL translation.
- SHACL is used for validating RDF knowledge graphs.
- The benchmark addresses the lack of dedicated benchmarks for NL2SHACL.
- Four state-of-the-art large language models were evaluated.
- Current LLMs generate syntactically valid SHACL but struggle with semantic equivalence for complex patterns.
- The paper is available on arXiv with identifier 2608.07530.
Entities
Institutions
- arXiv