Aymara AI Platform Evaluates 20 LLMs Across Safety Domains
A recent study has unveiled Aymara AI, a programmatic tool designed for the creation and management of tailored, policy-driven safety assessments for large language models (LLMs). This platform converts natural language safety guidelines into adversarial prompts and evaluates model outputs through an AI-based rater that has been validated against human assessments. The Aymara LLM Risk and Responsibility Matrix assesses 20 commercially available LLMs across 10 practical safety areas. Findings reveal significant performance variations, with average safety scores spanning from 86.2% to 52.4%. While models excelled in established areas like Misinformation (mean = 95.7%), they struggled in more intricate domains such as Privacy & Impersonation (mean = 24.3%). This research is available on arXiv.
Key facts
- Aymara AI is a programmatic platform for safety evaluation of LLMs.
- The platform transforms natural-language safety policies into adversarial prompts.
- An AI-based rater validated against human judgments scores model responses.
- The Aymara LLM Risk and Responsibility Matrix evaluates 20 commercially available LLMs.
- Evaluation covers 10 real-world safety domains.
- Mean safety scores range from 86.2% to 52.4%.
- Models scored 95.7% in Misinformation domain.
- Models scored 24.3% in Privacy & Impersonation domain.
Entities
Institutions
- Aymara AI
- arXiv