PeakPatch: Recovering Negation from CLIP's Frozen Features
There's been a major discovery about CLIP, the well-known vision-language model: it struggles to tell apart phrases that mean the opposite, like 'a dog' versus 'not a dog,' because they look so similar in its analysis. This problem, called Representational Collapse, happens because the model’s final layers focus more on visual matching than on the structure of the language. To fix this without retraining, researchers created a solution called PeakPatch. It works by using an Embedding Correction Network (ECN) that captures signals related to negation from earlier layers and then adds that back into the final output. This study, shared on arXiv (2607.23271), highlights a key flaw in contrastive models that could impact how we retrieve images and artwork when negation matters.
Key facts
- CLIP maps opposite phrases like 'a dog' and 'not a dog' to nearly identical embeddings.
- Representational Collapse occurs when final layers lose compositional syntax for visual alignment.
- Middle layers of CLIP's text encoder build compositional syntax.
- PeakPatch intercepts the encoder at its compositional peak.
- An Embedding Correction Network (ECN) uses cross-attention to extract negation-specific signals.
- PeakPatch is a lightweight post-hoc correction system.
- The paper is available on arXiv with ID 2607.23271.
- The method does not alter pretrained weights.
Entities
Institutions
- arXiv