OP-Mix: A Unified Data Mixing Algorithm for LLM Training
A new data mixing algorithm called OP-Mix (On-Policy Mix) has been introduced by researchers, designed to function throughout the entire training lifecycle of language models. This approach overcomes the shortcomings of current methods, which are limited to specific phases such as pretraining or continual learning. By framing data mixing as an online decision-making challenge, OP-Mix employs low-rank adapters to efficiently replicate potential mixtures. The research paper can be found on arXiv with the identifier 2605.15220.
Key facts
- OP-Mix is a data mixing algorithm for language model training
- It works across pretraining, continual learning, and adaptation phases
- Existing methods address only one phase at a time
- The approach treats data mixing as an online decision-making problem
- Candidate mixtures are simulated by interpolating low-rank adapters
- The paper is published on arXiv with ID 2605.15220
- The title is 'Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Time'
Entities
Institutions
- arXiv