RAG vs GraphRAG for Page-Level QA on Math Textbook
A research project assessed Retrieval-Augmented Generation (RAG) alongside GraphRAG for answering questions at the page level in an undergraduate mathematics textbook. The study utilized 477 question-answer pairs linked to particular pages, examining five embedding-based RAG models, a BM25 baseline, and GraphRAG for their retrieval precision and answer quality. The results indicated that embedding-based RAG surpassed GraphRAG, with voyage-3-large reaching an impressive 99.4% accuracy within the top-10. The BM25 baseline demonstrated robust performance, exceeding several embedding models. An error analysis revealed that 63.3% of top-1 mistakes involved retrieving content from the same chapter, highlighting its educational significance.
Key facts
- 477 question-answer pairs from an undergraduate math textbook
- Five embedding-based RAG models, BM25 baseline, and GraphRAG compared
- voyage-3-large achieved 99.4% retrieval accuracy at top-10
- BM25 outperformed several embedding models
- 63.3% of top-1 failures retrieved same-chapter content
- Study investigates LLMs as educational aids aligned with course materials
- Metrics: retrieval accuracy and answer quality (F1 score)
- GraphRAG underperformed embedding-based RAG for page-level retrieval
Entities
Institutions
- arXiv