GraphRAG for EHR Schema Retrieval Benchmarked on Consumer Hardware
A recent study investigated the effectiveness of Graph-based Retrieval Augmented Generation (GraphRAG) for accessing Electronic Health Record (EHR) schemas utilizing open-source large language models. The Microsoft GraphRAG system evaluated four models: Llama 3.1 (8 billion parameters), Mistral (7 billion), Qwen 2.5 (7 billion), and Phi-4-mini (3.8 billion), all operated on a single consumer-grade GPU with 8 GB of VRAM through the Ollama platform. The research focused on various metrics, including indexing efficiency, response accuracy, and challenges like cost and compliance issues in healthcare, especially where traditional cloud-based models may falter with intricate data requirements.
Key facts
- GraphRAG extends retrieval-augmented generation for structured reasoning over complex corpora.
- The study evaluates GraphRAG for EHR schema retrieval using local open-source LLMs.
- Four models benchmarked: Llama 3.1 (8B), Mistral (7B), Qwen 2.5 (7B), Phi-4-mini (3.8B).
- Models deployed via Ollama on a single consumer GPU (8 GB VRAM).
- Evaluation includes indexing efficiency, knowledge graph construction, query latency, answer quality, and hallucination.
- The work addresses cost, latency, and compliance challenges in healthcare.
- Real-world EHR schema documentation used as source material.
- Microsoft GraphRAG pipeline implemented.
Entities
Institutions
- Microsoft
- Ollama