CALVER: Symbolic Verification Boosts Causal Reasoning in LLMs
A recent paper published on arXiv (2608.03506) presents CALVER (Causal Axiom-Level VERification), a symbolic verifier that does not require training and aims to improve best-of-K causal reasoning in large language models (LLMs). It highlights a weakness in self-consistency techniques, where the most common response might be unreliable due to recurring confounding errors. CALVER evaluates structured traces based on Pearl's causal standards and chooses the candidate with the highest score. For CLEAR find-one-valid queries, CALVER achieves an accuracy of 42.1%, while competing methods hover around 30%. In a clean-core subset audit, 11 out of 21 selections made by CALVER differ from the benchmark's ground truth, indicating it may uncover valid answers missed by the benchmark. The paper is authored by researchers and is accessible on arXiv.
Key facts
- CALVER is a training-free symbolic verifier for causal reasoning in LLMs.
- It scores structured traces against Pearl's causal criteria: d-separation, backdoor adjustment, and intervention.
- On CLEAR find-one-valid queries, CALVER achieves 42.1% accuracy.
- Plurality, reward model, LLM judge, and model confidence remain near 30% on identical frozen pools.
- Scaling the judge to 72B does not close the performance gap.
- In an audited clean-core subset, 11 of 21 graph-valid CALVER selections differ from the benchmark's ground truth.
- The paper is available on arXiv with ID 2608.03506.
- CALVER does not consult a reference answer during selection.
Entities
Institutions
- arXiv