FairNVT Framework Enhances Fairness in Vision Transformers Through Noise Injection

A new debiasing framework called FairNVT has been introduced to improve fairness in pretrained transformer-based encoders while maintaining task accuracy. The approach addresses both representation and prediction level fairness simultaneously, arguing these aspects are inherently connected. By suppressing sensitive information at the representation level, the method facilitates fairer predictions downstream. FairNVT employs lightweight adapters to learn task-relevant and sensitive embeddings separately. Calibrated Gaussian noise is applied to the sensitive embedding before fusing it with the task representation. Orthogonality constraints and fairness regularization work together to reduce sensitive-attribute leakage in learned embeddings. The framework is compatible with various pretrained transformer encoders and has been tested across three datasets spanning vision and language domains. Unlike many existing approaches that treat representation and prediction fairness separately, FairNVT addresses them jointly through its integrated methodology. The paper detailing this framework was announced on arXiv with identifier 2604.16780v1.