DIVE: New Embedding Compression Method for LLMs
A new method called DIVE (Dimensionality reduction with Implicit View Ensembles) addresses overfitting in embedding compression for large language models. Proposed in arXiv:2605.20689, DIVE uses a self-limiting hinge-based triplet loss and head-wise NT-Xent contrastive loss to reduce dimensionality without degrading retrieval performance, unlike prior methods such as Matryoshka-Adaptor, Search-Adaptor, and SMEC. The approach bounds perturbations to pretrained embeddings, improving efficiency for vector search systems.
Key facts
- DIVE is a compression adapter for LLM embeddings.
- It uses self-limiting hinge-based triplet loss.
- It uses head-wise NT-Xent contrastive loss.
- Prior methods: Matryoshka-Adaptor, Search-Adaptor, SMEC.
- Overfitting occurs when labeled data is scarce.
- DIVE maintains retrieval performance above frozen baseline.
- Method reduces storage and computational costs.
- Paper appears on arXiv with ID 2605.20689.
Entities
Institutions
- arXiv