DEPT: A New Training Method for Unified Query Expansion and Retrieval in LLMs
A new paper on arXiv proposes Document Embedding Preservation Tuning (DEPT), a training approach for large language models that combines query expansion and document retrieval in a single decoder-only model. Traditional systems use separate modules or prompted expansions, which often fail to align with retrieval loss. DEPT tackles the moving-target problem, where updating the model for query expansion inadvertently changes the document embeddings used as retrieval targets. The method keeps tuned document embeddings close to their initial cached versions while allowing retrieval gradients to pass through straight-through decoding. This unified training enables the same model to generate expansions and encode both queries and documents. The paper, identified as arXiv:2608.17632, describes the technique in detail.
Key facts
- DEPT stands for Document Embedding Preservation Tuning.
- It trains a single decoder-only LLM end-to-end for query expansion and retrieval.
- Existing systems rely on prompted expansions, independent modules, or staged optimization.
- The moving-target problem arises when retrieval supervision shifts document embeddings.
- DEPT preserves document embeddings close to cached initial embeddings.
- Retrieval gradients pass through straight-through decoding.
- The paper is available on arXiv under identifier 2608.17632.
- The model encodes both expanded queries and candidate documents.
Entities
—