Automated Benchmark Generation Framework for Foundation Models
A newly developed framework for automated benchmark creation seeks to overcome the shortcomings of existing evaluation techniques for foundation models, which typically depend on aggregate scores from benchmarks that lack thorough coverage and metadata. This innovative system formulates evaluation challenges based on reference materials, such as textbooks, resulting in benchmarks that offer extensive coverage, detailed metadata, and resilience against contamination. It utilizes a multi-agent architecture for generating problems and a solution-graph-driven approach to enhance the reliability of ground truth. Three benchmarks were produced in Machine Learning, Corporate Finance, and Personal Finance. An expert review indicated a significantly reduced ground-truth error rate compared to earlier benchmarks like MMLU and GSM8K, with evaluations of 12 commercial and open-source models demonstrating nearly uniform competency coverage and performance.
Key facts
- Framework generates evaluation problems grounded in reference material like textbooks.
- Uses multi-agent architecture for problem generation.
- Employs solution-graph-driven strategy for ground truth reliability.
- Three benchmarks generated: Machine Learning, Corporate Finance, Personal Finance.
- Lower ground-truth error rate than MMLU and GSM8K.
- Evaluated 12 commercial and open-source models.
- Achieves near-uniform competency coverage.
- Benchmarks are robust to contamination.
Entities
—