New AI Research Proposes Few-Shot Preference Optimization Method for LLM Personalization

Researchers have introduced Few-Shot Preference Optimization (FSPO), a novel algorithm designed to personalize large language models (LLMs) by reframing reward modeling as a meta-learning problem. The approach enables LLMs to quickly infer personalized reward functions for individual users using only a few labeled preference examples. To address the challenge of collecting real-world preference data at scale, the team constructed synthetic preference datasets, generating over 1 million synthetic personalized preferences using publicly available LLMs. FSPO incorporates user description rationalization (RAT) to improve both reward modeling and instruction-following capabilities, helping recover performance comparable to using an oracle user description. This research, documented in arXiv preprint 2502.19312v2, aims to enhance personalization for user-facing applications like virtual assistants and content curation systems. The work demonstrates how synthetic data can be effectively transferred to real-world scenarios through careful design choices. Personalization remains critical for broad adoption of LLM-based interfaces across various domains.