Reinforcement Learning for Code Optimization Faces Measurement Noise and Reward Sparsity
A recent preprint on arXiv (2607.25970) tackles the difficulties associated with applying reinforcement learning (RL) to optimize code. Although RL has proven effective in ensuring code correctness by incentivizing programs that succeed in hidden tests, integrating execution time into the reward system for optimization presents notable challenges. The authors highlight three primary issues: measurement noise, reward sparsity, and GRPO instability, which hinder RL's effectiveness, resulting in marginally improved solutions with increased failures. To address these challenges, they suggest a three-part strategy: (1) developing DMC-Optim, a benchmark featuring extensive optimization tests and a calibrated sandbox for dependable evaluation; (2) merging correctness and speed rewards in the RL framework, utilizing an offline simulator for optimal configuration predictions; and (3) modifying GRPO and assessment for the more challenging timed-execution environment. This research aims to make execution time a learnable factor in RL-based code optimization.
Key facts
- arXiv:2607.25970 addresses RL for code optimization.
- RL for code correctness is established by rewarding programs that pass hidden tests.
- Extending RL to optimization by adding execution time to reward causes problems.
- Three key problems: measurement noise, reward sparsity, GRPO instability.
- Generated solutions are barely faster and more fail when timing drives reward.
- Three-stage approach: DMC-Optim benchmark, reward composition with offline simulator, adapted GRPO.
- DMC-Optim includes large optimization tests and a calibrated sandbox.
- Offline simulator predicts most promising configurations for speed reward.
Entities
Institutions
- arXiv