AdaMTP: Adaptive Training Paradigm for Multi-Token Prediction
A new adaptive training framework known as AdaMTP has been developed for Multi-Token Prediction (MTP) within large language models. MTP enhances a common backbone by adding auxiliary heads that enable simultaneous predictions of multiple future tokens, thereby improving supervision and speeding up inference. Current models, however, utilize a static prediction horizon, neglecting the varying information density found in natural language and code. This limitation compels auxiliary heads to make predictions across complex semantic boundaries, introducing disruptive noise that hampers essential functions. AdaMTP addresses this issue by adjusting the prediction horizon to match the sequence's inherent predictability, employing an entropy-based segmentation technique that utilizes the base model. The research can be found on arXiv with the identifier 2608.00434.
Key facts
- AdaMTP is an adaptive training paradigm for Multi-Token Prediction (MTP).
- MTP augments a shared Large Language Model backbone with auxiliary heads.
- Existing training frameworks use a fixed-length prediction horizon.
- Fixed horizons disregard non-uniform information density in language and code.
- Predicting across high-entropy semantic boundaries injects noisy training signals.
- AdaMTP dynamically aligns prediction horizon with sequence predictability.
- An entropy-based segmentation algorithm detects predictability.
- Paper available on arXiv:2608.00434.
Entities
Institutions
- arXiv