AVISE Framework for AI Security Evaluation

A novel open-source framework named AVISE (AI Vulnerability Identification and Security Evaluation) has been launched to methodically pinpoint vulnerabilities and assess the security of AI systems and models. This framework enhances the multi-turn Red Queen attack, based on theory-of-mind, by incorporating an Adversarial Language Model (ALM) augmented attack. Additionally, it features an automated Security Evaluation Test (SET) designed to uncover jailbreak vulnerabilities in language models. The SET consists of 25 test cases along with an Evaluation Language Model (ELM) that evaluates whether each case successfully jailbroke the target model, achieving a remarkable 92% accuracy and an F1-score. This research is available on arXiv with the identifier 2604.20833.