Legal RAG Systems Still Hallucinate Frequently, Study Finds

A recent investigation published on arXiv (paper 2608.14210) offers an in-depth examination of hallucination tendencies in eight retrieval-augmented generation (RAG) systems within two legal datasets: the English GDPR and a French national civil law. The study employed both claim-level and answer-level assessments to evaluate the frequency and intensity of hallucinations, while also considering various question types and user profiles. Findings were corroborated using a separate collection of 142 questions authored by legal experts. The results indicate that hallucinations are widespread, with the best systems showing under 10% of responses affected, while the worst approached nearly 50%. Particularly, questions based on false premises led to elevated hallucination rates. The research underscores the ongoing issue of ungrounded responses in legal AI, where implications can be significant. The paper can be accessed on arXiv under identifier 2608.14210.