Niyam-AI: Provable AI Safety via Zero-Knowledge Proofs
The newly introduced framework, Niyam-AI, seeks to ensure the safety of AI agents through cryptographic verification. At the beginning of each session, it secures allowed tools and constraints within an Intent Contract, which is committed using SHA-256. An isolated Judge model intercepts and authenticates each tool call, generating a zk-SNARK proof via EZKL upon successful validation. The tool is executed only after this proof is verified, enabling external parties to validate enforcement without needing access to the Judge model's weights. This framework tackles issues like prompt injection, hallucinated reasoning, and unsafe tool usage—key vulnerabilities for autonomous LLM agents. Niyam-AI was tested against 2,000 real-world scenarios from an undisclosed source. The related paper can be found on arXiv under identifier 2608.07167.
Key facts
- Niyam-AI is a framework for provable AI safety enforcement.
- It uses an Intent Contract committed via SHA-256.
- A Judge model validates tool calls.
- zk-SNARK proofs are generated via EZKL.
- Tool execution occurs only after proof verification.
- Third parties can verify enforcement without accessing Judge weights.
- Evaluated on 2,000 real-world scenarios.
- Paper available on arXiv (2608.07167).
Entities
Institutions
- arXiv