Fast Byte Latent Transformer Speeds Up Byte-Level Language Models
A new paper on arXiv introduces techniques to accelerate byte-level language models (LMs), which match token-level performance without subword vocabularies but suffer from slow autoregressive generation. The Byte Latent Transformer (BLT) is enhanced with BLT Diffusion (BLT-D), trained with a block-wise diffusion objective alongside next-byte prediction, enabling parallel byte generation per decoding step. Two extensions inspired by speculative decoding, BLT Self-speculation (BLT-S) and another, trade speed for quality by having the local decoder draft bytes beyond normal patch boundaries. The paper is authored by researchers and posted on arXiv with ID 2605.08044.
Key facts
- Byte-level LMs match token-level performance without subword vocabularies.
- BLT Diffusion (BLT-D) is a new model variant trained with a block-wise diffusion objective.
- BLT-D generates multiple bytes in parallel per decoding step.
- BLT Self-speculation (BLT-S) extends speculative decoding to BLT.
- BLT-S's local decoder continues generating past normal patch boundaries to draft bytes.
- The paper is available on arXiv under ID 2605.08044.
- The techniques aim to reduce the number of forward passes required for generation.
- The paper proposes two extensions inspired by speculative decoding.
Entities
Institutions
- arXiv