MIPO: Mutual Information Preference Optimization enhances LLM personalization without extra data

Researchers propose Mutual Information Preference Optimization (MIPO), a self-improvement framework for large language models (LLMs) that requires no additional human-labeled data or external verifiers. MIPO constructs preference pairs by generating a positive response conditioned on the correct prompt and a negative response conditioned on a random, unrelated prompt. Using Direct Preference Optimization (DPO) to learn from these pairs maximizes pointwise conditional mutual information between prompts and model responses under the base LLM. This approach addresses the limitation of post-training methods that rely on expensive human oversight and existing exploited data. The method is detailed in arXiv preprint 2603.19294v2.