Q-CARE: A New Framework for Query-Agnostic RAG Evaluation
A novel framework named Q-CARE has been introduced to evaluate retrieval-augmented generation systems, as outlined in a recent paper on arXiv. This framework aims to deliver consistent assessments for diverse user inquiries, from simple fact inquiries to intricate queries. Q-CARE breaks down questions and answers into smaller, verifiable components, employing metrics like coverage-aware retrieval metrics and claim-level assessments such as completeness and verifiability. In tests involving a human-annotated benchmark across eight datasets, Q-CARE demonstrated superior alignment with human evaluations compared to four existing metrics. This development marks a significant advancement in assessing the factual accuracy of RAG systems.
Key facts
- Q-CARE is a query-agnostic and fully reference-free RAG evaluation framework.
- It decomposes queries into sub-queries and answers into atomic claims.
- It introduces coverage-aware retriever metrics: C-Prec@k and C-nDCG@k.
- It introduces claim-level generator metrics: Completeness, Conciseness, and Verifiableness.
- Q-CARE was evaluated on a human-annotated benchmark spanning eight datasets.
- Q-CARE achieved higher correlation with human judgments than RAGEval and RAGChecker.
- The framework aims to improve factuality assessment of large language models in RAG systems.
- The paper is available on arXiv with ID 2608.11238.
Entities
Institutions
- arXiv