Syntax-Informed Positional Embeddings Enhance Transformer Performance

A recent paper published on arXiv presents Syntax-informed Positional Embeddings (SiPE), which incorporates syntactic structures into the positional embeddings of Transformer models. This research, accessible at arXiv:2608.06111, reveals that SiPE acquires a lightweight syntactic prior from dependency parses during pretraining, applying it across three main families of positional embeddings: absolute, relative, and rotary, for both encoders and decoders. The architecture and self-attention mechanisms remain unchanged. The authors identified optimal integration points for the prior, showing that for autoregressive decoders with relative positional embeddings, a multiplicative coupling with the relative-position term of the attention score yields the best results. For encoders, the prior should be added directly to input embeddings. This cross-type announcement was released on August 26, 2026, indicating that integrating syntactic information can enhance Transformers' performance in tasks requiring sentence structure comprehension.