IRIS: A New Self-Play Fine-Tuning Framework for LLMs

A novel framework for self-play fine-tuning, named IRIS (Interpolative Rényi Iterative Self-play), has been introduced for large language models. This approach enhances models beyond traditional supervised fine-tuning, eliminating the need for human annotations by comparing annotated outputs with those generated by the model itself. Current techniques such as SPIN (KL-based), SPACE (Jensen-Shannon through noise contrastive estimation), and SPIF (χ²-regularized) each excel in different scenarios based on the distributional gap between the model and the target; however, no single divergence achieves optimal learning dynamics throughout all training phases. IRIS employs a Rényi-based objective with a flexible order parameter α, dividing into two independent tilted risk components for annotated and synthetic data, with exponential importance weights managed by α. This framework seeks to effectively balance learning from both real and generated datasets. The research can be found on arXiv under ID 2604.20933.