FaithformBench: New Benchmark for Faithful Autoformalisation in Lean
A new benchmark called FaithformBench has been developed by researchers to evaluate the faithfulness of autoformalisation (AF) systems, which transform natural language reasoning into formal statements for proof assistants such as Lean. This benchmark overcomes the shortcomings of current evaluation techniques that typically depend on costly human-annotated ground truths or LLM judges, along with embedding models that have limited accuracy assurances. FaithformBench is economical, operates under minimal assumptions, and assesses both valid and invalid examples by automatically creating perturbed reasoning steps that are deliberately incorrect. It analyzes the preservation of validity in unperturbed steps and the preservation of invalidity in perturbed ones. The method was tested on eight AF systems, though specific outcomes are not included in the abstract. This research is available on arXiv, identified as 2608.10916.
Key facts
- FaithformBench is a new benchmark for autoformalisation faithfulness.
- It assesses both positive and negative examples.
- The method uses automatically generated perturbed reasoning steps.
- It measures validity preservation on unperturbed steps and invalidity preservation on perturbed steps.
- The benchmark is cheap to apply and sound under weak assumptions.
- It was applied to eight AF systems.
- The paper is available on arXiv (2608.10916).
- The work focuses on proof assistants such as Lean.
Entities
Institutions
- arXiv