SupChain-Bench: Benchmarking LLMs for Supply Chain Management
SupChain-Bench is an innovative benchmark aimed at assessing large language models (LLMs) in practical supply chain management scenarios. It evaluates domain expertise and the ability to orchestrate tools over extended periods, based on standard operating procedures (SOPs). Research indicates notable deficiencies in execution reliability among existing models. Additionally, the creators introduce SupChain-ReAct, a framework that operates without SOPs, autonomously generating executable procedures for tool utilization and demonstrating superior and consistent performance in tool-calling. This initiative sets a foundational benchmark for examining dependable long-horizon orchestration within supply chain processes.
Key facts
- SupChain-Bench is a unified real-world benchmark for supply chain management.
- It evaluates LLMs on domain knowledge and long-horizon tool-based orchestration.
- Experiments show substantial gaps in execution reliability across models.
- SupChain-ReAct is an SOP-free framework that synthesizes executable procedures.
- SupChain-ReAct achieves the strongest and most consistent tool-calling performance.
- The benchmark is grounded in standard operating procedures (SOPs).
- The work aims to study reliable long-horizon orchestration.
- The paper is available on arXiv with ID 2602.07342.
Entities
Institutions
- arXiv