ASSERT: A New Pipeline for Auditing Generative AI Systems
A recent paper on arXiv (ID: 2608.13840) has unveiled a new auditing measurement pipeline for generative AI (GenAI) systems, referred to as ASSERT. This pipeline seeks to clarify the confusion surrounding the compliance rates reported for GenAI systems by linking each rate to a detailed specification of the measurement criteria utilized. By adopting this specification-driven method, researchers can create behavioral rubrics and test cases, subsequently conducting audits on GenAI systems to yield a compliance rate. In a case study focused on conversational deception, findings revealed that the reported rate fluctuates significantly based on factors like dialogue setup, simulated user, judge, and evidence threshold for non-compliance. This underscores the necessity for clear measurement choices in AI audits, as variations in reported rates may stem from either the system's actions or the auditing process. This paper is a cross-submission announcement and can be found on arXiv.
Key facts
- ASSERT is a specification-driven measurement pipeline for GenAI audits.
- It ties each reported rate to a written specification of measurement choices.
- The pipeline helps draft a behavioral rubric and test cases.
- It runs the audit against a GenAI system and returns a reported rate.
- A case study on conversational deception showed that the reported rate varies with dialogue setup, simulated user, judge, and evidence bar.
- The paper is available on arXiv with ID 2608.13840.
- The announcement type is cross.
- The research addresses the issue of unclear whether changes in reported rates are due to system or measurement choices.
Entities
Institutions
- arXiv