SciRet: Compute-Aware Study of Retrieval and Reranking for Scientific RAG
A recent study titled SciRet evaluates the effectiveness of retrieval-augmented generation (RAG) for answering scientific inquiries using the CORD-19 dataset. The research experimented with a standardized RAG approach across three dataset sizes: 1,034, 5,160, and 15,480 chunks, representing 1,000, 5,000, and 15,000 papers, respectively. The investigation found that integrating multiple retrieval techniques yields better results than single-method approaches, achieving a perfect Recall@10 for both the smallest and largest datasets. However, the application of a cross-encoder reranker trained with MS MARCO negatively impacted accuracy in scientific scenarios. Notably, larger datasets contributed to more reliable generation outcomes.
Key facts
- SciRet is a compute-aware empirical study of retrieval-augmented generation for scientific question answering.
- The study uses the CORD-19 corpus.
- Three corpus scales are evaluated: 1,034 chunks (1K papers), 5,160 chunks (5K papers), and 15,480 chunks (15K papers).
- The pipeline includes sentence-window chunking, BM25, BGE-M3 dense retrieval, reciprocal rank fusion, optional cross-encoder reranking, and grounded answer generation.
- Hybrid retrieval reaches Recall@10 of 1.000 at 1K and 15K scales.
- An MS MARCO-trained cross-encoder reranker reduces precision on the scientific corpus.
- Generation faithfulness measured with RAGAS increases with corpus scale.
- The paper is available on arXiv with identifier 2608.03860.
Entities
Institutions
- arXiv