HiPO: Hierarchical Preference Optimization Enhances LLM Reasoning

A new paper on arXiv introduces HiPO (Hierarchical Preference Optimization), an extension of Direct Preference Optimization (DPO) for aligning large language models with human preferences in complex reasoning tasks. DPO optimizes entire responses but lacks granular feedback on multi-step solutions. Existing methods like KTO, RSO, ReMA, and Tree of Thoughts excel in either stable preference learning or structured reasoning, but not both. HiPO separates responses into reasoning segments—query clarification, reasoning steps, and answer—and computes a weighted sum of DPO losses per segment, enabling segment-specific training while maintaining DPO's computational efficiency.