Byte-Prefix Marginalization Enables Cross-Tokenizer On-Policy Distillation
A new technique called Byte-Prefix Marginalization (BPM) has been developed by researchers for on-policy distillation between language models utilizing different tokenizers. This method reformulates the next-token distribution from the teacher model to align with the student's vocabulary by associating teacher token probabilities with the longest student token that shares a prefix in byte representation. While the aggregated mass for identical student tokens is maintained, any unmatched mass is categorized into a distinct residual group. This results in a target that is complete in vocabulary, byte-aligned, and preserves mass for dense on-policy distillation. BPM overcomes the challenges of traditional full-vocabulary on-policy distillation, which usually relies on a common tokenizer, and enhances existing cross-tokenizer approaches that might lose probability mass or misassign it. The full details can be found in arXiv preprint 2607.22334.
Key facts
- BPM stands for Byte-Prefix Marginalization.
- It enables on-policy distillation across different tokenizers.
- Teacher token probabilities are mapped to student tokens via byte prefix matching.
- Unmatched probability mass is placed in a residual category.
- The target distribution is vocabulary-complete and mass-preserving.
- Existing cross-tokenizer methods may discard or misassign probability mass.
- Full-vocabulary OPD typically assumes a shared tokenizer.
- The method is described in arXiv:2607.22334.
Entities
Institutions
- arXiv