Novelty and Surprise Prioritized Experience Replay: A New Method for Sample-Efficient Image-Based RL
arXiv preprint 2608.17373 introduces NSPER (Novelty and Surprise Prioritized Experience Replay), a method for enhancing sample efficiency in image-based reinforcement learning. The approach combines novelty, which identifies underrepresented states, with surprise, which reveals gaps in the agent's understanding of its environment. It builds on Prioritized Experience Replay (PER), which reuses high-value transitions, and intrinsic rewards, which encourage exploration. Conventional sampling strategies select experiences unpredictably, resulting in redundant updates and slower learning. The authors propose NSPER to address this issue and also present NSPER+R, a further extension that integrates these signals. The work aims to prioritize informative experiences and improve learning speed in visual domains.
Key facts
- Sample efficiency is a central challenge in reinforcement learning, especially in image-based domains.
- Traditional sampling often relies on random or suboptimal experience selection, leading to redundant updates and slow learning.
- Prioritized Experience Replay (PER) reuses high-value transitions.
- Intrinsic rewards promote exploration of novel or uncertain states.
- The integration of PER and intrinsic rewards has not been extensively studied.
- The paper introduces Novelty and Surprise Prioritized Experience Replay (NSPER).
- NSPER uses novelty to capture underrepresented states and surprise to expose gaps in the agent's understanding.
- The paper introduces NSPER+R, which integrates these signals.
Entities
—