Hierarchical RL Optimizes Diffusion Multi-Modal LLMs
A new reinforcement learning method, Hierarchical Token GRPO (HT-GRPO), addresses optimization challenges in Diffusion Multi-Modal Large Language Models (dMLLMs). The approach introduces a Sketch-Then-Paint training scheme with three stages—global, structure, and refinement—to account for the hierarchical generation process where early tokens define layout and later tokens add details. This method improves policy optimization by assigning rewards based on token contribution, overcoming issues with intractable importance ratios in existing RL techniques. The paper is available on arXiv (2605.16842).
Key facts
- HT-GRPO integrates hierarchical generation into policy optimization for dMLLMs.
- Sketch-Then-Paint scheme has three stages: global, structure, refinement.
- Existing RL methods assign uniform rewards ignoring token contribution.
- Single image can be generated via many unmasking sequences, complicating RL.
- Early tokens define global layout; later tokens focus on local details.
- Paper published on arXiv with ID 2605.16842.
- Method addresses intractable importance ratios in current RL approaches.
- HT-GRPO stands for Hierarchical Token GRPO.
Entities
Institutions
- arXiv