LLM Task Adaptation's Uneven Impact on Alignment Studied
A recent study has examined the effects of various post-training methods on the alignment of large language models (LLMs). Three methods were evaluated: Supervised Fine-Tuning (SFT), KL-regularized SFT, and Reinforcement Learning via Reward (RLVR). The research assessed 15 alignment aspects across six domains, revealing that RLVR enhances task performance with minimal alignment changes, while SFT leads to greater alignment drift. In contrast, KL regularization effectively mitigates drift. The study is accessible on arXiv under paper ID 2607.22676.
Key facts
- Study evaluates post-training methods on LLM alignment
- Methods tested: SFT, KL-regularized SFT, and RLVR
- 15 alignment aspects across 6 domains
- RLVR improves task performance with small alignment shifts
- SFT causes larger alignment drift
- KL regularization reduces drift
- Study available on arXiv
- Paper ID: 2607.22676
Entities
Institutions
- arXiv