Vertex-Softmax: Tighter Transformer Verification via Exact Optimization
A new method called Vertex-Softmax achieves the tightest possible sound bound for verifying transformer attention mechanisms under interval constraints on pre-softmax scores. The approach proves that the exact optimum of the score-box problem occurs at a vertex of the constraint box, and a threshold structure theorem reduces candidate optima to linearly many after sorting objective coefficients, yielding log-linear complexity in sequence length. Integrated into a CROWN-style verifier, Vertex-Softmax establishes a formal optimality result, showing that further improvement requires additional structure like score correlations or score-value coupling. The paper is available on arXiv under identifier 2605.10974.
Key facts
- Vertex-Softmax is a new primitive for transformer verification.
- It achieves the tightest sound bound from score intervals alone.
- The exact optimum of the score-box problem is at a vertex of the constraint box.
- A threshold structure theorem reduces candidate optima to linearly many.
- Complexity is log-linear in sequence length.
- Formal optimality result shows need for additional structure for improvement.
- Integrated into a CROWN-style verifier.
- Paper available on arXiv: 2605.10974.
Entities
Institutions
- arXiv