CPR Prevents Policy Collapse in Continual RL
A new optimizer called Calibrated Partial Resets (CPR) prevents policy collapse in continual reinforcement learning. Neural networks suffer from dormant neurons and loss of expressivity in non-stationary settings. Full resets sacrifice performance and destabilize training. CPR periodically pulls low-utility neurons toward initialization, with pull strength scaled by utility. Unlike binary resets or uniform decay, CPR avoids brittleness and concentrates adjustment where needed. In SlipperyAnt, only CPR avoids collapse over 400M training steps.
Key facts
- CPR stands for Calibrated Partial Resets
- CPR is an optimizer for continual reinforcement learning
- Neural networks accumulate dormant neurons in non-stationary settings
- Full unit reinitialization can cause policy collapse
- CPR pulls low-utility neurons toward initialization
- Pull strength is scaled by each neuron's utility
- Only CPR avoids policy collapse over 400M training steps in SlipperyAnt
- CPR outperforms compared methods
Entities
—