LLMs Struggle with Long-Chain Reasoning on Equivalence Class Problem

A new empirical study on arXiv evaluates Large Language Models (LLMs) on the Equivalence Class Problem (ECP), a simple but long-chain reasoning task. The study tests both reasoning and non-reasoning models across varying variables, connectivity probabilities, and prompts. Non-reasoning LLMs fail ECP entirely, while reasoning models perform significantly better but still cannot fully solve it. For non-reasoning models, performance drops as connectivity probability increases with fixed variables. The paper highlights fundamental limitations in LLM reasoning capabilities.