RAG-TESTER: Automated End-to-End Testing of Retrieval-Augmented Large Language Models
06:00 · August 4, 2026 · arXiv cs.AI RSS
Retrieval-Augmented Generation (RAG) enables Large Language Models (LLMs) to use external and domain-specific knowledge, but its reliability depends on the interaction between the generative model, embedding model, retrieval mechanism, and prompt construction strategy. We present RagTester, an automated end-to-end testing approach for RAG systems. RagTester generates retrieval documents, test inputs, and expected outputs; executes the tests; and evaluates the resulting answers using an LLM as a judge. Its test-generation strategy targets complex passages, unsupported queries, and document-coverage criteria. We evaluate RagTester using eight LLMs and six embedding models, yielding 24 compatible configurations, and compare it with a baseline test-input generator. Across 72,000 test executions, RagTester detected 21,633 failures, 6.6% more than the baseline, and outperformed it in 20 of the 24 configurations. The detected failures include inaccurate retrieval, unsupported answers, incomplete use of retrieved context, and difficulties interpreting complex passages. These results show that coverage-oriented test generation can effectively expose failures caused by the interaction between retrieval and generation components and support the assessment of RAG configurations before deployment.
Summary
RAG-TESTER provides an automated framework for end-to-end testing of retrieval-augmented generation systems. It addresses the fact that RAG reliability emerges from interactions among the generative model, embedding model, retrieval mechanism, and prompt construction, rather than from any single component. The approach proceeds through four stages: automatic generation of retrieval documents, creation of test inputs and expected outputs that target complex passages, unsupported queries, and document-coverage criteria, execution of those tests against a chosen RAG configuration, and evaluation of the resulting answers by an LLM acting as judge.
In a large-scale study the framework was applied to eight LLMs paired with six embedding models, producing 24 distinct configurations. Three thousand test inputs were generated and executed across all pairings for a total of 72,000 runs. Compared with a baseline test-input generator, RAG-TESTER identified 21,633 failures—an increase of 6.6 percent—and performed better in 20 of the 24 configurations. The failures it surfaced included inaccurate retrieval, answers unsupported by retrieved context, incomplete use of that context, and difficulties interpreting complex passages.
These outcomes indicate that coverage-oriented test generation can expose interaction faults that remain hidden under narrower evaluation methods. By supplying both the test-generation logic and an automated oracle, the tool enables systematic comparison of LLM–embedding combinations before deployment. The authors have released the implementation and replication package to support further use in assessing RAG reliability.
Why it matters
Provides actionable, coverage-oriented testing methodology directly applicable by Dutch AI teams deploying RAG in enterprise or regulated settings; aligns with EU emphasis on transparent, reliable AI and supports pre-deployment validation of LLM configurations.
