ZCA Whitening Improves WEAT Bias Measurement in Language Models
A new paper on arXiv (2608.06908) presents a technique called Zero-phase Component Analysis (ZCA) whitening, aimed at improving the Word Embedding Association Test (WEAT). This test is widely used to evaluate bias in AI and social sciences by measuring semantic associations through cosine similarity. However, many language models haven’t met the assumption of an isotropic embedding space, which has cast doubt on bias evaluations. ZCA whitening adjusts the covariance to resemble an identity matrix while keeping the original vectors largely intact, thus restoring isotropy. The authors tested this method on ten WEAT test suites and seven models from three different architectures, totaling 70 combinations, and found that ZCA whitening significantly boosts WEAT score reliability. The paper hints at a potential prior publication as well.
Key facts
- Paper arXiv:2608.06908 proposes ZCA whitening as a pre-processing step for WEAT.
- WEAT is a bias measurement method used in computational social science and AI fairness research.
- WEAT relies on cosine similarity, assuming isotropy of the embedding space.
- Many language models do not satisfy the isotropy assumption, affecting reliability.
- ZCA whitening transforms covariance to identity matrix while minimizing perturbation.
- Evaluation covers ten WEAT test suites and seven models from three architectural families.
- Total of 70 model-task combinations were tested.
- Results show ZCA whitening improves WEAT reliability.
Entities
Institutions
- arXiv