PromptEmbedder: Dual-LLM Framework for Efficient Text Embedding
A team of researchers has introduced PromptEmbedder, a dual-LLM framework designed to separate embedding knowledge from specific backbone weights, facilitating efficient architectural transfers. This system employs a Prompting LLM to create instruction-aware soft prompts for a static Embedding LLM through differentiable generation with continuous relaxation, allowing complete gradient flow during contrastive training. To adapt to new backbones, only a lightweight linear alignment matrix needs retraining, thus eliminating the need for expensive retraining from the ground up. Evaluations conducted on the MTEB benchmark indicate performance levels similar to LoRA. This research has been published on arXiv.
Key facts
- PromptEmbedder is a dual-LLM framework for text embedding.
- It decouples embedding knowledge from specific backbone weights.
- Uses a Prompting LLM to generate soft prompts for a frozen Embedding LLM.
- Differentiable generation with continuous relaxation ensures gradient flow.
- Adaptation to new architectures requires only retraining a linear alignment matrix.
- Evaluated on MTEB benchmark, achieves comparable performance with LoRA.
- Addresses bottlenecks in computational efficiency and cross-architecture transferability.
- Published on arXiv with ID 2605.28066.
Entities
Institutions
- arXiv