Full-Bandwidth Transformer: Latent Feedback for Autoregressive Models

A recent study published on arXiv (2608.08888v1) presents the full-bandwidth transformer, an upgrade for autoregressive transformers. This research tackles a drawback found in conventional transformer models: although dense attention allows extensive horizontal access to prior tokens, the vertical feedback between decoding steps is limited, as only the sampled token returns to the stack's base, discarding the top-layer hidden state. The new approach enhances this feedback channel by incorporating latent feedback; at each decoding step, the previous top-layer hidden state merges with the sampled token embedding via a gated linear unit, which is then reintegrated as the next input. This method maintains the original transformer structure, KV cache, and language-modeling goal while employing a scheduled multi-pass objective for training. The full paper can be accessed at https://arxiv.org/abs/2608.08888.