arXiv Paper Proposes Intentional Updates Method for Streaming Reinforcement Learning

A recent paper on arXiv presents a novel method called intentional updates to tackle instability in streaming reinforcement learning. This technique begins by defining the desired result of an update and then determining the step size that can closely achieve this goal. Unlike conventional gradient-based methods, where step sizes in parameter units lead to unpredictable changes in function output, this approach is more structured. The instability issue is especially pronounced in streaming scenarios with a batch size of 1, where randomness is not averaged out, leading to potentially extreme update magnitudes. The paper adapts this concept for streaming deep reinforcement learning by establishing clear intended outcomes: Intentional TD seeks a consistent fractional reduction of the TD error, while Intentional Policy Gradient targets a limited per-step change. This method draws inspiration from the Normalized Least Mean Squares algorithm used in online supervised linear regression, which selects step sizes to ensure specified changes in output based on current error. The paper, identified as 2604.19033v1, was categorized as a cross announcement on arXiv.