XAI Evaluation Card Proposed to Standardize Metrics Reporting
A recent publication on arXiv introduces the XAI Evaluation Card, a template aimed at standardizing the documentation of evaluation metrics for explainable AI (XAI). The authors highlight transparency as a significant concern, noting that metrics are frequently poorly defined, inadequately reported, and unvalidated against established baselines. This card mandates a clear statement of target properties, grounding levels, assumptions regarding metrics, validation proof, risks of manipulation, and recognized failure instances. Embracing this template as a standard within the community may help diminish fragmentation in evaluations, facilitate meta-analyses, and enhance accountability in XAI research.
Key facts
- Paper proposes XAI Evaluation Card for standardizing XAI metric reporting
- Identifies lack of standardization in XAI evaluation
- Metrics are inconsistently defined and incompletely reported
- Rarely validated against common baselines
- Card covers target properties, grounding levels, assumptions, validation, gaming risks, failure cases
- Aims to reduce evaluation fragmentation
- Supports meta-analysis and accountability
- Published on arXiv
Entities
Institutions
- arXiv