FrontierOR Benchmark Tests LLMs on Large-Scale Optimization

FrontierOR has been launched by researchers as a benchmark to assess large language models (LLMs) in their ability to design efficient algorithms for complex large-scale optimization challenges. This benchmark features 180 tasks sourced from leading operations research publications, each accompanied by standardized instances and a concealed, expert-validated evaluation framework. Seven LLMs, including cutting-edge, cost-efficient, and open-source models, were tested in both one-shot and test-time evolution scenarios. Findings indicate that current LLMs face difficulties in scalable algorithm design, frequently underperforming compared to direct formulation-and-solve methods. FrontierOR seeks to enhance LLM performance in operations research by emphasizing the exploitation of problem structures and scalability, tackling the shortcomings of existing benchmarks that focus on smaller or simplified cases.