Deep RL Fails to Beat Calibrated Baselines in Adaptive Resource Control

A new benchmark study challenges the effectiveness of deep reinforcement learning (DRL) for adaptive resource control. The paper introduces RLScale-Bench, a reproducible benchmark evaluating six mainstream DRL algorithms—PPO, DQN, A2C, SAC, TD3, and DDPG—against a calibrated rule-based autoscaler. Across six workload patterns and five seeds (240 runs), the rule-based controller achieved the lowest cost on all workloads, though it trailed the best RL agents on bursty and flash traffic. The benchmark is instantiated on Kubernetes Horizontal Pod Autoscaling and probes distribution-shift generalization. The study finds that discrete-action DRL methods perform poorly, and that the calibrated baseline is often overlooked in prior work. The paper is available on arXiv under ID 2605.26418.