HiLP: Hierarchical Latent Prediction Enhances Long-Horizon Reasoning in Language Models
Researchers have introduced Hierarchical Latent Prediction (HiLP), a novel training objective for language models that aims to improve long-horizon reasoning and planning. The method, detailed in a paper on arXiv, addresses limitations of standard Next-Token Prediction (NTP) and recent alternatives like Multi-Token Prediction (MTP) and Next-Latent prediction (NextLat). HiLP introduces an auxiliary higher-level abstract latent to reduce error accumulation in latent-space rollouts, leading to longer-horizon coherent belief state representation. Experiments demonstrate effectiveness across coding and multi-step reasoning benchmarks, and offer more speculative decoding efficiency. The paper is available on arXiv with the identifier 2608.05806.
Key facts
- HiLP introduces an auxiliary higher-level abstract latent to reduce error accumulation in latent-space rollouts.
- The method is designed to improve long-horizon reasoning and planning in language models.
- It addresses limitations of standard Next-Token Prediction (NTP) and recent alternatives like Multi-Token Prediction (MTP) and Next-Latent prediction (NextLat).
- Experiments show HiLP leads to longer-horizon coherent belief state representation.
- The method is effective across coding and multi-step reasoning benchmarks.
- HiLP offers more speculative decoding efficiency.
- The paper is available on arXiv with identifier 2608.05806.
- The paper is categorized under Computer Science > Computation and Language.
Entities
Institutions
- arXiv