smevals: A New Tool for Evaluating AI Models, Prompts, and Harnesses
Simon Willison has released smevals, a new evaluation framework developed in collaboration with Jesse Vincent's Prime Radiant applied AI research lab. The tool is designed to run small evaluation suites across different model configurations and grade the results. It allows users to create evals as directories containing YAML files, run them against models, and grade the runs using defined checks. Results can be explored via a localhost web server or built into static HTML reports. Willison describes smevals as his third iteration on the idea, noting that figuring out the vocabulary was the most time-consuming part. The announcement was made on 31st July 2026, and the tool is intended to help answer questions about model capabilities. Willison also mentions plans to expand the tool and apply it to his own projects. The post includes a link to the tool and a sponsor option for a monthly digest of LLM developments.
Key facts
- smevals is a new evaluation framework for AI models, prompts, and harnesses.
- Developed with Jesse Vincent's Prime Radiant applied AI research lab.
- Evals are created as directories with YAML files.
- Runs are separate from grading operations.
- Results can be viewed via a localhost web server or static HTML.
- Willison calls it his third iteration on the idea.
- Announced on 31st July 2026.
- Willison plans to expand the tool and use it in his own projects.
Entities
Institutions
- Prime Radiant