Personalized Observation Normalization for Heterogeneous Federated RL
A novel approach known as personalized observation normalization (PON) tackles the issue of heterogeneity in federated reinforcement learning (FedRL). In this framework, several agents develop a unified policy without exchanging raw data; however, varied environments lead to differing state-transition dynamics and uneven parameter updates. PON enables each agent to locally standardize raw state inputs by utilizing a continuously updated running mean and variance, ensuring uniform scaling that avoids dominance during aggregation. The ineffectiveness of sharing normalization parameters among agents stems from their distinct local input distributions, underscoring the necessity for tailored statistics. Experiments conducted on diverse MuJoCo tasks validate the effectiveness of this method.
Key facts
- Federated reinforcement learning enables collaborative training without sharing raw data.
- Heterogeneous environments cause non-identical input distributions and imbalanced updates.
- PON normalizes state inputs locally using running mean and variance.
- Sharing normalization parameters across agents is ineffective.
- Experiments conducted on heterogeneous MuJoCo tasks.
- Paper published on arXiv with ID 2605.27385.
- Method ensures consistent scaling without overshadowing during aggregation.
- Personalized statistics are necessary due to diverse local input distributions.
Entities
Institutions
- arXiv