Equivariance Constraints Can Reduce Neural Network Expressive Power
A recent study published on arXiv examines how enforcing equivariance constraints affects the expressive capabilities of 2-layer ReLU networks. The researchers illustrate that while these constraints may limit expressive power, increasing the model size can offset this limitation. They establish upper limits on the necessary model expansion and reveal that larger architectures result in a lower dimensionality of the hypothesis space, which suggests improved generalization.
Key facts
- Paper focuses on 2-layer ReLU networks
- Enforcing equivariance constraints can undermine expressive power
- Drawback can be compensated by enlarging model size
- Upper bounds on required enlargement are proven
- Enlarged architectures have reduced hypothesis space dimensionality
- Implies better generalizability
- Published on arXiv
- Submission history available
Entities
Institutions
- arXiv