AdaFRUGAL: Dynamic Memory Optimization for LLM Training

AdaFRUGAL introduces dynamic controls for memory-efficient training of large language models, automating hyperparameter tuning that previously required manual intervention. The method extends the FRUGAL framework by incorporating a linear decay for the subspace ratio (ρ) and a loss-aware schedule for update frequency (T). Experiments on English C4 and Vietnamese VietVault pre-training datasets, as well as GLUE fine-tuning, show AdaFRUGAL maintains competitive performance against AdamW and static FRUGAL while reducing GPU memory and training time. This offers a practical solution for resource-constrained environments.