AISPA: Auditing System Prompts in 88 Commercial AI Products
A new framework called Artificial Intelligence System Prompt Assurance (AISPA) has been introduced to audit system prompts in AI applications. System prompts are developer-configured instructions that govern AI model behavior but are rarely disclosed, creating trust and accountability gaps. AISPA evaluates prompts along eight user-relevant dimensions and was applied to 3,249 instructions from 88 commercial AI products. The audit found that prompt design varies widely, with some organizations averaging over 60 protective instructions per product. The framework classifies instructions as protective or problematic, aiming to enhance transparency and user protection in AI deployment.
Key facts
- AISPA is a user-centric framework for auditing system prompts.
- The framework evaluates prompts along eight dimensions.
- The audit reviewed 3,249 instructions from 88 commercial AI products.
- Instructions are classified as protective or problematic.
- System prompt design varies substantially across products and developers.
- Some organizations average over 60 protective instructions per product.
- System prompts are rarely disclosed to the public or regulators.
- The framework aims to address trust and accountability gaps in AI systems.
Entities
Institutions
- arXiv