Reason Popper-ly: Neurosymbolic Framework for LLM Reasoning
Researchers have introduced Reason Popper-ly, a neurosymbolic framework that leverages inductive logic programming (ILP) to derive rules for relation composition from reasoning traces. This framework functions as an online verifier for step-level corrections in large language models (LLMs). When provided with a chain-of-thought trace generated by an LLM, the approach evaluates each inferred step against a learned rule table, identifies types of violations, corrects erroneous steps using symbolically derived modifications, and regenerates the remaining suffix to ensure the model yields a final answer based on a verified trace. The framework was tested on CLUTRR, a multi-hop kinship reasoning benchmark, utilizing five language models across reasoning chains ranging from 2 to 10 hops, consistently enhancing terminal accuracy compared to standard chain-of-thought prompting.
Key facts
- Reason Popper-ly uses inductive logic programming (ILP) to learn relation composition rules from reasoning traces.
- It acts as an online verifier for step-level correction in LLM chain-of-thought reasoning.
- The method checks each inferred step against a learned rule table.
- It diagnoses violation types and rewrites incorrect steps with symbolically derived repairs.
- The remaining suffix is regenerated conditioned on a verified trace.
- Evaluated on CLUTRR benchmark using five language models.
- Reasoning chains ranged from 2 to 10 hops.
- Consistently improves terminal accuracy over standard chain-of-thought prompting.
Entities
—