RagTester: Automated End-to-End Testing for RAG Systems
A novel automated testing method for Retrieval-Augmented Generation (RAG) systems has been presented in a paper on arXiv, identified as 2608.00054. This technique, called RagTester, creates retrieval documents, test inputs, and anticipated outputs, subsequently running tests and assessing responses with an LLM serving as the evaluator. Its strategy for generating tests focuses on intricate passages, unsupported queries, and criteria for document coverage. The evaluation process included eight LLMs and six embedding models, resulting in 24 viable configurations, which were compared to a baseline test-input generator. Over 72,000 test runs, RagTester identified 21,633 failures—6.6% more than the baseline—and excelled in 20 out of 24 configurations. The failures noted pertain to both retrieval and generation issues.
Key facts
- RagTester is an automated end-to-end testing approach for RAG systems.
- It generates retrieval documents, test inputs, and expected outputs.
- It uses an LLM as a judge to evaluate answers.
- Test-generation strategy targets complex passages, unsupported queries, and document-coverage criteria.
- Evaluation used eight LLMs and six embedding models, yielding 24 configurations.
- 72,000 test executions were performed.
- RagTester detected 21,633 failures, 6.6% more than the baseline.
- It outperformed the baseline in 20 of 24 configurations.
Entities
Institutions
- arXiv