Budgeted Attention Allocation Boosts Transformer Efficiency

A novel technique named Budgeted Attention Allocation enables transformers to function across various cost-quality levels using a single trained model. This method incorporates a monotone head-gating system based on the specified attention budget. The importance of dense warm-starting for stability was highlighted. In a synthetic sequence task, the budgeted model recorded an impressive 99.7% accuracy at an estimated attention cost of 0.303 and achieved 100.0% at a cost of 0.504. For AG News, a custom word-level transformer with hard-gate adaptation attained 82.1% accuracy and a 1.28x speedup at a budget of 0.50. Additionally, budgeted structural pruning on a pretrained BERT-Mini yielded 87.6% accuracy with a 1.20x speedup at the same budget, surpassing a zero-shot dense post-hoc baseline (86.1%) and closely approaching a per-budget specialist after one recovery epoch (87.9%). The approach was also evaluated on DBpedia14 using BERT-Mini.