Transformers Can Simulate Arbitrary Attention Mechanisms

A new paper on arXiv investigates whether transformer encoders can simulate arbitrary attention mechanisms. The authors construct a universal simulator U composed of transformer encoder layers that can replicate the computations of any vanilla attention mechanism. This work sits at the intersection of learnability and expressivity, addressing a theoretical gap between data-driven probabilistic guarantees and deterministic computability proofs. Previous research established Turing-completeness for transformers and explored bounds on circuit complexity and formal logic. The study provides a theoretical framework for understanding the computational limits of transformer architectures.