New Loss Family for Tuning GFlowNets and LLMs
A study has been released that presents a collection of surrogate loss functions aimed at training generative models, such as GFlowNets and large language models (LLMs). The researchers demonstrate that the mean square error, which measures the difference between target and model log probabilities and has been an effective low-variance loss, can be generalized to encompass the entire set of f-divergences. This innovative family of losses exhibits the characteristic that, when assessed on-policy, their gradients align with those of the respective f-divergence, while maintaining the same global minimizer off-policy. The research establishes a direct link between translation invariant loss functions on log probabilities and f-divergences, facilitating the creation of new surrogate losses for optimizing various generative models. The paper is available on arXiv with the identifier 2605.15417.
Key facts
- The paper introduces a family of surrogate loss functions for training generative models.
- The loss family extends the mean square error between target and model log probabilities.
- On-policy gradients of the new losses correspond to those of f-divergences.
- Off-policy, the losses retain the same global minimizer.
- The work establishes a one-to-one correspondence between translation invariant loss functions and f-divergences.
- The loss family can be applied to GFlowNets, generative models, and LLMs.
- The paper is published on arXiv with identifier 2605.15417.
- The research enables new surrogate losses for tuning a wide class of generative models.
Entities
Institutions
- arXiv