SIRIN: New Toolkit Detects Contextual Hallucinations in LLM Systems

A new toolkit named SIRIN (Semantic Inconsistency Recognition and Inspection Nexus) has been developed by researchers to identify contextual hallucinations in retrieval-augmented, agentic, and memory-grounded LLM systems. Contextual hallucinations refer to coherent and plausible answers that lack backing from the given evidence. SIRIN integrates three detection approaches—representation probing, uncertainty estimation, and judge-style verification—along with pre-generation query answerability into a single interface, configuration system, and evaluation pipeline. It allows for both response- and span-level analysis in white-box and black-box environments. The interactive web UI facilitates real-time examination of context-query-answer triples, featuring hallucination scores and unsupported-span highlighting. This system is showcased for its effectiveness in detecting hallucinations and assessing query answerability. The research can be found on arXiv with the identifier 2608.00033.