MARS: Adaptive Data Augmentation Framework for Reward Modeling in RLHF

Researchers have introduced a new framework called MARS, which stands for Margin and Semantic-Aware Data Augmentation for Reward Modeling, aimed at boosting reward modeling in scenarios with limited data. Reward modeling is crucial in techniques like reinforcement learning from human feedback (RLHF) and AI feedback (RLAIF), but it often struggles with insufficient and inconsistent human preference data. MARS tackles this by emphasizing augmentation for low-margin preference pairs and enhancing semantic distinctions to effectively separate accepted from rejected responses before generating synthetic preference samples. Evaluated across three datasets and two reward-modeling frameworks, MARS showed better performance in RewardBench and alignment win rates compared to traditional methods. The full details are available on arXiv under the identifier 2602.17658 in the Machine Learning section.