Retraction Normalizations on Hyperspheres Unify Residual Connections and GeoNorm
A recent publication on arXiv (2608.02668) presents a comprehensive framework for interpreting residual connections in deep neural networks through Riemannian geometry. The researchers reveal that the set of retraction maps on a hypersphere simplifies to a singular scalar design choice, which dictates the transformation of an update's magnitude into a rotation angle. This framework unifies Euclidean residual connections with Geodesic Normalization (GeoNorm) within the same theoretical framework. By applying metric projection and Cayley retractions, the authors introduce Proj-Sphere, a novel normalization technique. This work is classified as a cross-announcement and is accessible on arXiv.
Key facts
- Paper title: 'Sphere Retraction Normalizations'
- arXiv ID: 2608.02668
- Announcement type: cross
- GeoNorm recasts residual connections on a Riemannian manifold
- GeoNorm orthogonalizes each layer output against the current hidden state
- The update is applied via the Riemannian exponential map
- Every hidden state maintains a constant l2-norm, confining the residual stream to a hypersphere
- The exponential map is one member of a broad family of retraction maps
- On the hypersphere, the entire family collapses to a single scalar design choice
- The distinction between retractions is how update magnitude is converted into a rotation angle
- The framework unifies Euclidean residual connections and GeoNorm
- Instantiations with metric projection and Cayley retractions yield Proj-Sphere
Entities
Institutions
- arXiv