Mask-Based Priors Outperform Query-Key Initializations in Transformer Boolean Extrapolation

A recent study published on arXiv (2608.00418) explores the shortcomings of Transformers in Boolean extrapolation tasks, revealing that they frequently generalize based on a simpler minimum-degree interpolator instead of the actual target function. The researchers argue that incorporating explicit structural priors into attention mechanisms could address this issue. They observe that current structured-initialization techniques, which modify inductive bias by selecting query and key projections to mimic a specific attention pattern, are quickly overwritten during the training process, failing to alter the learned extrapolation rule. The authors advocate for mask-based priors as a more durable solution. Although the study employs controlled stress tests to analyze Transformer inductive bias, the tasks do not represent practical applications.