DEFault++: Automated Fault Detection for Transformer Architectures
Transformer models play a crucial role in essential AI applications; however, issues within their attention mechanisms, projections, and other internal components can lead to performance degradation without triggering any runtime errors. Current fault diagnosis methods primarily focus on general deep neural networks and fail to pinpoint the specific transformer component linked to a particular symptom. This article introduces DEFault++, a diagnostic approach based on hierarchical learning that functions at three abstraction levels: it detects the presence of faults, categorizes them into one of 12 transformer-specific fault types (including attention-internal mechanisms and related architectural components), and determines the root cause from a pool of up to 45 mechanisms. To support training and evaluation, we developed DEFault-bench, a benchmark comprising 3,739 labeled instances derived from systematic mutation testing.
Key facts
- DEFault++ is a hierarchical learning-based diagnostic technique for transformer architectures.
- It operates at three levels: detection, classification into 12 fault categories, and root cause identification from up to 45 mechanisms.
- The technique addresses faults in attention mechanisms, projections, and other internal components.
- Existing techniques target generic DNNs and cannot identify specific transformer components responsible for faults.
- DEFault-bench is a benchmark of 3,739 labeled instances created through systematic mutation testing.
- The paper is published on arXiv with ID 2604.28118.
- Faults in transformers often degrade behavior silently without runtime errors.
- The 12 fault categories cover attention-internal mechanisms and surrounding architectural components.
Entities
Institutions
- arXiv