SeGaBench: Benchmarking LLMs to Recover Compiler-Optimization Semantics

There's a new study on arXiv that introduces something called SeGaBench. It's a benchmark designed to evaluate how well large language models (LLMs) can spot semantic optimization chances that compilers might miss. It features 100 synthetic scenarios and another 20 based on actual source code, focusing on various aspects like low-level assumptions and data-structure rules. Each scenario comes with hidden semantics, an oracle artifact, and validators to ensure accuracy. The study looks at five different LLMs, collecting five responses for each case. The best model got the correct artifacts in 94.8% of cases, managed at least a 1.05x speedup in 83.3%, and was accurate in 93.3%. However, it only partially addressed the oracle gap, indicating that LLMs don’t completely match the oracle's performance. You can find the paper under the identifier 2608.03983 on arXiv.