ReLU-Based Method Boosts Transformer Verification Precision
A new approach to formal verification of transformers improves precision by using ReLU to represent non-linear bounds for dot products. Transformers, widely deployed in safety-critical applications, involve complex computations like self-attention that make verification difficult. Existing over-approximation methods using convex constraints are efficient but introduce approximation errors leading to false alarms. The proposed method leverages ReLU to precisely bound dot products, enabling more accurate verification while maintaining efficiency.
Key facts
- Formal verification of transformers is increasingly important for safety-critical applications.
- Transformer inferences involve complex computations like dot products in self-attention layers.
- Existing over-approximation methods use convex constraints but sacrifice precision.
- The new approach uses ReLU to represent precise non-linear bounds for dot products.
- The method aims to reduce false alarms caused by approximation errors.
- The paper is published on arXiv with ID 2605.14294.
Entities
Institutions
- arXiv