Weak-to-Strong Alignment Analyzed via Bias-Variance Lens

A recent study available on arXiv delves into the transition from weak to strong alignment in artificial intelligence systems. The investigation highlights instances where robust models exhibit confidence in incorrect decisions that less capable teachers cannot recognize. Researchers propose a new framework, incorporating bias-variance-covariance concepts to link misfit theory with post-training outcomes. They establish a misfit-based threshold for population risk during the enhancement process. To evaluate this, the study examines continuous confidence measurements and tests four methodologies: supervised fine-tuning, reinforcement learning from human feedback, and reinforcement learning from AI feedback, utilizing the PKU-SafeRLHF and HH-RLHF datasets.