Impossibility of Faithful, Stable, and Complete Feature Ranking Under Collinearity
A recent study published on arXiv demonstrates that it is impossible for any feature ranking technique to be simultaneously faithful, stable, and complete in the presence of collinear features. In cases of collinearity, rankings become random. This limitation is illustrated across four different model types: for gradient boosting, the attribution ratio diverges as 1/(1-rho^2), is infinite for Lasso, and converges in random forests. The authors address this challenge through ensemble averaging with DASH (Diversified Aggregation of SHAP), which is shown to be Pareto-optimal among unbiased aggregations and meets the Cramer-Rao variance limit with a precise ensemble size formula. The analysis identifies two distinct families: faithful-complete methods (unstable, with rankings changing up to 50% of the time) and stable ensemble methods like DASH, which report ties for symmetric features. The findings are verified by 305 theorems in Lean 4.
Key facts
- No feature ranking can be simultaneously faithful, stable, and complete under collinearity.
- For collinear pairs, ranking reduces to a coin flip.
- Impossibility quantified for four model classes: gradient boosting, Lasso, random forests, and others.
- Attribution ratio diverges as 1/(1-rho^2) for gradient boosting.
- Attribution ratio is infinite for Lasso.
- Attribution ratio converges for random forests.
- DASH (Diversified Aggregation of SHAP) resolves the impossibility via ensemble averaging.
- DASH is provably Pareto-optimal among unbiased aggregations.
- DASH achieves the Cramer-Rao variance bound with a tight ensemble size formula.
- Two families of methods exist: faithful-complete (unstable) and ensemble methods like DASH (stable).
- Faithful-complete methods have rankings that flip up to 50% of the time.
- Ensemble methods report ties for symmetric features.
- The impossibility is machine-verified with 305 Lean 4 theorems.
Entities
Institutions
- arXiv