Advanced Multimodal RAG for Scientific Document Understanding
A novel framework introduced in arXiv paper 2607.24799 presents an Advanced Multimodal Retrieval-Augmented Generation (RAG) architecture designed to enhance the extraction of information from scientific literature. This method tackles significant obstacles in domain-specific question answering, including constrained contextual knowledge, variances between sparse and dense retrieval methods, and retrieval inaccuracies. Utilizing the open-source Vision-Language Model Qwen2-VL-2B-Instruct, the multimodal ingestion pipeline creates textual summaries for tables and figures. For the retrieval process, it combines HNSW-based semantic search with GIN-based lexical search, harmonized through Reciprocal Rank Fusion and fine-tuned with Cross Encoder reranking to reduce noise. The objective is to decrease hallucination in Large Language Models without prior fine-tuning, thereby improving accuracy in understanding scientific documents.
Key facts
- arXiv paper 2607.24799 introduces an Advanced Multimodal RAG system.
- System uses Qwen2-VL-2B-Instruct to summarize tables and figures.
- Retrieval combines HNSW semantic search and GIN lexical search.
- Reciprocal Rank Fusion unifies search results.
- Cross Encoder reranking reduces retrieval noise.
- Aims to reduce hallucination in LLMs for scientific documents.
- Addresses limited context knowledge and retrieval noise.
- No prior fine-tuning required for domain-specific questions.
Entities
Institutions
- arXiv