FAST-GOAL Enhances CLIP for Long Text Descriptions

Researchers have introduced FAST-GOAL (Fast and Efficient Global-local Object Alignment Learning), a fine-tuning method to improve CLIP's handling of lengthy text descriptions. CLIP, a vision-language model, struggles with detailed text due to its pre-training on short captions. FAST-GOAL employs two components: Fast Local Image-Sentence Matching (FLISM) extracts local image regions via object detection and spatial division, matching them with sentences; Token Similarity-based Learning (TSL) maximizes similarity between patch tokens from specific image regions and their region embeddings, applying the same to text. The method enhances the model's ability to capture detailed correspondences. The paper is available on arXiv.