Lightweight Chunk Selection for Mobile RAG Systems
A new research paper on arXiv (2608.03148) proposes a lightweight method for selecting relevant chunks in retrieval-augmented generation (RAG) systems, specifically designed for mobile and edge devices. The approach treats chunk selection as an evidence-alignment problem, combining three complementary feature sources: question hidden states, mixture-of-experts (MoE) routing signals, and retrieved chunk embeddings. This method aims to reduce computational and memory costs associated with RAG deployment on resource-constrained devices, addressing the challenge that top-ranked chunks are not always the most evidence-supporting. The paper is categorized as a cross-type announcement and is available at the provided arXiv URL.
Key facts
- Paper arXiv:2608.03148 proposes lightweight chunk selection for mobile RAG.
- RAG improves factual grounding of LLMs by incorporating external knowledge.
- Deploying RAG on mobile devices is challenging due to increased computation and memory.
- Direct cost reduction method retains only one retrieved chunk, but top-ranked chunk may not be most evidence-supporting.
- Existing context-reduction methods often require additional LLMs or compressors, costly under mobile budget.
- Selector combines question hidden states, MoE routing-derived expert signals, and retrieved chunk embeddings.
- The approach is framed as an evidence-alignment problem.
- The paper is announced as a cross-type on arXiv.
Entities
Institutions
- arXiv