Tree Generation Method Reduces Forgetting in Large Language Models

Researchers have introduced a model-agnostic self-decompression method called Tree Generation (TG) to address catastrophic forgetting in Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs). LLMs often forget previously learned knowledge when fine-tuned on domain-specific data, while MLLMs like LLaVA show performance declines on language benchmarks compared to their single-modality versions. The TG method decompresses knowledge within LLMs into the training corpus, generating synthetic supervised fine-tuning (SFT) data for instruction tuning. By incorporating this dumped corpus during SFT for MLLMs, the forgetting problem is significantly reduced. The paper, titled "Preserving Knowledge in Large Language Model with Model-Agnostic Self-Decompression," focuses on TG-SFT and was submitted to arXiv on June 17, 2024.