On-Policy Distillation Boosts Compact ASR Models with Limited Data

Researchers have introduced Ark-ASR, a language model with 0.6 billion parameters that is audio-conditioned and trained on just 100,000 hours of speech. This model employs on-policy distillation from the more robust Qwen-ASR teacher. Ark-ASR consistently surpasses the performance of supervised fine-tuning alone and exceeds the Qwen3-ASR-0.6B baseline across four out of five ASR benchmarks in Mandarin and English. Notably, it utilizes 100,000 hours of speech, while the Qwen3-Omni AuT encoder relies on 20 million hours. Although the larger Qwen3-ASR-1.7B model remains superior, these findings indicate that teacher-guided on-policy training can significantly enhance compact ASR models with a much lower audio resource investment.