RAVEN-Eval: Rubric-Guided Automated Evaluation for AI Video Generation Models

A new framework called RAVEN-Eval has been introduced to address the challenge of evaluating advanced AI video generation models (AIVGMs). As AI-generated videos become more sophisticated and widely used commercially, traditional metrics like visual fidelity and semantic instruction following are no longer sufficient to distinguish quality differences among top-tier models. Human evaluation, meanwhile, has become more costly and demanding, requiring expert annotators and sustained attention. RAVEN-Eval, presented in a paper on arXiv (2608.09111), is a rubric-guided automated evaluation framework built on the LMM-as-a-judge paradigm. It employs an automatic task curation and quality-filtering pipeline to assemble a dataset of 150 text-to-video (T2V) and 100 image-to-video (I2V) tasks. The framework systematically collects evaluations from large multimodal models (LMMs) acting as judges, guided by rubrics to ensure consistent and fine-grained assessments. This approach aims to minimize human intervention while providing reliable differentiation between advanced AIVGMs. The paper details the framework's methodology and its potential to streamline evaluation processes in the rapidly evolving field of AI video generation.