CaRE: A Compute-Aware Evaluation Protocol for Masked Diffusion Language Models
A recent study presents CaRE, a framework designed for the evaluation of compute-aware masked diffusion language models (MDLMs). The researchers contend that existing evaluation methods fall short, as seven recent remasking studies utilize inconsistent settings, differing in step counts, metrics, and sampling temperatures, which undermines the reliability of comparisons. CaRE addresses this by standardizing the number of function evaluations (NFE), mandating multi-metric reporting, and managing stochasticity. When applied to seven remasking techniques on LLaDA-8B-Base and Dream-7B-Base across four levels of stochasticity and three step budgets using OpenWebText and LM1B, CaRE indicates that the reported improvements may stem from evaluation artifacts instead of true algorithmic advancements.
Key facts
- CaRE is a compute-aware evaluation framework for MDLMs.
- Seven recent remasking papers use incompatible evaluation settings.
- CaRE standardizes NFE, enforces multi-metric reporting, and controls stochasticity.
- Applied to LLaDA-8B-Base and Dream-7B-Base.
- Tested on OpenWebText and LM1B datasets.
- Evaluates 7 remasking strategies at 4 stochasticity levels and 3 step budgets.
- Reported gains may be evaluation artifacts.
- Paper is on arXiv with ID 2607.24763.
Entities
Institutions
- arXiv