Q-Learning Error Analysis with Sign Separation

A new study that just came out on arXiv presents an error analysis for Q-learning with a constant step size, focusing on how positive and negative errors behave differently. It uses a switching-system approach, where negative errors are constrained by a linear time-invariant system tied to a fixed optimal policy, while positive errors are managed by a linear switching system. This research reveals that there's a significant imbalance in the error dynamics due to overestimations—specifically, the Bellman maximum is capable of amplifying positive errors, whereas negative ones face restrictions. Additionally, the study provides finite-time limits that are relevant for both deterministic and stochastic situations involving constant step sizes.