Frozen Random CNNs Develop Sparse Representations in Deep RL
A new study on arXiv reveals that when deep reinforcement learning agents use frozen, randomly set CNN feature extractors, they tend to develop extremely sparse fully-connected representations without any focus on promoting sparsity. In the first fully-connected layer (FC1, from 3,136 down to 64), these agents utilize just 1-3 neurons for predictable Pong and 5-11 for unpredictable Pong. In contrast, trainable CNNs typically activate 55-64 neurons under similar scenarios. The findings highlight that FC1 sparsity varies with task difficulty—1-11 for Pong, 19-26 for Breakout, and close to 42 for Space Invaders. Interestingly, three identical seeds in Pong result in 5, 7, and 11 active neurons, leading to different performance results, suggesting that random features can effectively capture essential task information through natural sparsity.
Key facts
- Deep RL agents with frozen random CNNs develop sparse FC1 representations without sparsity-inducing objectives.
- FC1 sparsity ranges from 1-3 neurons for deterministic Pong, 5-11 for stochastic Pong.
- Trainable CNNs activate 55-64 neurons under matched conditions.
- FC1 sparsity scales with task complexity: 1-11 for Pong, 19-26 for Breakout, ~42 for Space Invaders.
- Width-scaling confirms sparsity reflects task structure, not fixed capacity fraction.
- Three identical Pong seeds yield 5, 7, and 11 active neurons.
- The 5-neuron seed plateaus at +14 reward, while others reach +18.4 expert performance.
- Study published on arXiv with ID 2607.26059.
Entities
Institutions
- arXiv