Efficient LAM Evaluation Achieves 0.93 Correlation with Full Benchmarks Using 0.3% Data

Researchers propose using minimal subsets of just 50 examples (0.3% of data) to evaluate large audio models (LAMs), achieving over 0.93 Pearson correlation with full benchmark scores. The study analyzed 10 subset selection methods across 18 audio models and 40 tasks. To align with user satisfaction, 776 human preference ratings were collected from realistic voice assistant conversations, revealing only 0.85 correlation between both subsets and full benchmarks with human preferences. Regression models trained on selected subsets achieved 0.98 correlation, outperforming those trained on full data. The findings suggest efficient evaluation can reduce costs while maintaining reliability.