ADAPT: Online Reweighting Improves LLM Data Curation Over Offline Methods
A new arXiv paper proposes ADAPT (Adaptive Data reweighting for Pretraining and FineTuning), an online framework for data curation in LLM training. Unlike offline methods that detach curation from training via static selection or mixing, ADAPT dynamically adjusts sample importance through loss weighting during training. It uses adaptive per-sample learning rates guided by similarity-based quality signals, preserving data diversity and improving generalization. The approach addresses engineering overhead and brittleness of offline pipelines that require re-running under model or task shifts. The paper is available at arXiv:2605.05227.
Key facts
- ADAPT is an online reweighting framework for LLM training data curation.
- It uses adaptive per-sample learning rates guided by similarity-based quality signals.
- Offline methods detach curation from training, causing engineering overhead and brittleness.
- ADAPT dynamically adjusts sample importance via loss weighting during training.
- The approach preserves data diversity and improves generalization.
- Paper available at arXiv:2605.05227.
- Data curation is critical yet under-explored in LLM training.
- Offline methods alter data size through hard filtering or resampling.
Entities
Institutions
- arXiv