Hyperball Optimizers' Advantage Remains Unclear

A recent study published on arXiv questions the presumed benefits of Hyperball-style optimizers in scale-invariant deep networks. The researchers introduce an angular effective learning rate that incorporates the angle of parameter updates, as well as the norms of parameters and updates. They demonstrate that the traditional norm-based approach is merely a specific instance in cases of orthogonality. By breaking down updates into radial and tangential parts, they discover that radial updates have minimal direct impact on the angular effective learning rate, which fails to clarify why MuonH experiences slower convergence compared to MuonWD during the initial training phases.