Generalized Gaussian TD Error for Uncertainty-Aware RL
A novel reinforcement learning technique substitutes the traditional zero-mean Gaussian model for temporal difference (TD) errors with a Generalized Gaussian Distribution (GGD), enhancing the modeling of heavy-tailed and heteroscedastic residuals arising from bootstrapping and exploration. This method incorporates a state-conditioned shape head that utilizes GGD and a numerically adjusted GGD loss to serve as an online proxy for nonstationary TD residuals. It differentiates between the precise GGD likelihood (normalized for all β>0) and its exponential density kernel (which is positive definite for β∈(0,2]). A straightforward monotone weighting heuristic is formulated based on the learned shape, alongside Batch Inverse Error Variance (BIEV) regularization that employs variance and sample excess kurtosis of ensemble TD errors. Results from benchmarks on both continuous and discrete control tasks indicate that shape-aware variants surpass Gaussian-based approaches.
Key facts
- Conventional TD learning assumes zero-mean Gaussian errors.
- Heavy-tailed and heteroscedastic residuals arise from bootstrapping and exploration.
- Proposed method uses Generalized Gaussian Distribution (GGD) for shape head.
- GGD likelihood is normalized for every β>0.
- Exponential density kernel is positive definite for β∈(0,2].
- Learned shape enables a monotone weighting heuristic.
- Batch Inverse Error Variance (BIEV) regularization uses variance and sample excess kurtosis.
- Shape-aware variants outperform Gaussian-based methods on control benchmarks.
Entities
—