Open ASR Leaderboard Adds Private Datasets to Combat Benchmaxxing

The Open ASR Leaderboard, launched in September 2023 and visited over 710K times, has introduced private datasets from Appen Inc. and DataoceanAI to mitigate benchmaxxing and test-set contamination. These high-quality English ASR datasets cover scripted and conversational speech across multiple accents. To prevent exploitation, the datasets are kept private, and the leaderboard's default Average WER remains computed on public datasets only, with an optional toggle to include private data. The leaderboard emphasizes standardization through a normalizer based on Whisper that removes punctuation and casing and maps to American spelling, and openness via open-sourced UI code and evaluation scripts. New metrics include macroaverages for scripted, conversational, US, and non-US accents, without providing per-split scores to discourage optimization. Model developers can submit models via GitHub pull requests, with results verified on public sets and private metrics computed separately. The initiative aims to provide a more holistic view of ASR performance, acknowledging that no single model excels across all dimensions. Future plans include evaluations reflecting real-world noisy conditions.