SolidCoder: LLM Code Generation via Concrete Execution

SolidCoder is an innovative framework created to address the Mental-Reality Gap in code generation by LLMs, where these models often produce incorrect execution traces and validate flawed code. This gap includes two main issues: the Specification Gap, which overlooks edge cases, and the Verification Gap, where models incorrectly assert that faulty code behaves correctly. SolidCoder uses property-based oracles to replace imaginary traces with actual sandboxed execution and stresses the importance of considering edge cases before designing algorithms. With GPT-4o, it achieves remarkable pass rates: 95.7% on HumanEval (up 0.6%), 77.0% on CodeContests (up 4.3%), and 26.7% on APPS (up 3.4%). The core idea here is "don't imagine — execute."