Nexusformer Introduces Nonlinear Attention for Scalable Transformer Architecture

A novel transformer architecture named Nexusformer has been introduced to tackle the scaling challenges faced by conventional models. Standard transformers necessitate the training of larger variants from the ground up due to their attention mechanisms relying on linear projections, which restrict feature extraction to fixed-dimensional subspaces. This limitation hampers both expressivity and the ability to incrementally expand capacity. Nexusformer innovatively substitutes these linear Q/K/V projections with a Nexus-Rank layer, utilizing a three-stage nonlinear mapping activated by dual activations in increasingly higher dimensional spaces. This approach eliminates linearity restrictions and facilitates lossless structured growth. New capacity can be added along two axes through zero-initialized blocks that maintain pretrained knowledge. Experiments indicate that Nexusformer achieves Tokenformer's perplexity while consuming up to 41.5% less training compute. The findings were published on arXiv under the identifier arXiv:2604.19147v1.