BayesAME: Bayesian Framework for Automatic Coreset Size Determination in Model Evaluation

BayesAME is a sequential Bayesian framework designed to automatically determine the optimal coreset size for evaluating large generative models. Current methods require practitioners to manually specify the number of items to evaluate, but BayesAME prioritizes reliable performance estimation by inferring the necessary subset size from data. It models performance as a random variable, defining a latent ability for groups of items that share historical model performances, and uses a joint prior distribution to encode the belief that the target model behaves similarly to past models. The posterior distribution over these latent abilities enables automatic coreset selection, balancing efficiency and accuracy. This approach addresses the computational cost of evaluating large models on full benchmarks, offering a principled alternative to fixed-size coresets.