Multi-Byte Prediction Speeds Up Hierarchical Language Models

A new research paper on arXiv (2608.15454) introduces multi-byte prediction (MBP), a method to accelerate inference in byte-level hierarchical language models. These models, which generate text byte by byte, have emerged as a robust alternative to subword tokenization-based models, but their inference speed is a bottleneck. MBP generates multiple bytes in parallel, improving speed with minimal performance impact and no additional parameters. It builds on multi-token prediction (MTP) with two key innovations: a variable-length prediction window aligned with the latent tokens (segments) of the hierarchical model, and a novel attention-masking scheme that allows parallel byte prediction without violating causality. The authors demonstrate that MBP achieves a Pareto-optimal trade-off across generative tasks including instruction following, question answering, summarization, and machine translation. The paper is announced as a new submission and is available at the provided URL.