Transformer Architecture Derived from Spherical Geometry

A recent study entitled 'RT-Transformer: The Transformer Block as a Spherical State Estimator' reveals that the fundamental elements of the Transformer block—attention, normalization, and residual connections—naturally arise from a singular geometric estimation challenge. The researchers conceptualize the latent state as a direction on a hypersphere, with noise characterized in the tangent plane at the current estimate. This approach leads to a precision-weighted directional inference method where attention consolidates evidence, residual connections facilitate incremental updates to the state, and normalization retracts the revised state onto the hypersphere. The authors contend that these components stem from the geometry of the estimation issue, rather than being separate architectural decisions. This work is available on arXiv in the Computer Science > Machine Learning section.