TLA$^{+}$-Bench: Execution-Grounded Benchmark for NL-to-TLA$^{+}$ Specification Generation

A new dataset and benchmark called TLA$^{+}$-Bench has been developed by researchers to assess the capability of large language models in producing TLA$^{+}$ formal specifications from descriptions in natural language. Unlike earlier datasets that evaluated based on similarity to a reference or parsing success, TLA$^{+}$-Bench assesses through execution: each gold specification is paired with a configuration that the TLA$^{+}$ model checker utilizes to explore the entire reachable state space, conclusively verifying if the specification meets the specified properties. The dataset comprises 403 model-checked gold specifications and 897 parse-only silver specifications from 13 public repositories, encompassing previous TLA$^{+}$ generation datasets. Additionally, it features four descriptions written by models in two styles from two sources, complete with difficulty and category labels. A significant observation is that an exact oracle produces a range of correctness metrics based on the evaluation method used.