Interpretability-Guided Data Selection Boosts LLM Fine-Tuning Efficiency

A new framework called Interpretability-Guided Data Selection (IGDS) has been introduced by researchers. This approach employs mechanistic interpretability tools, such as Sparse Autoencoders (SAEs), to pinpoint causal features related to tasks in Large Language Models (LLMs) and to choose 'Feature-Resonant Data' for fine-tuning. Initially, IGDS identifies task features through frequency recall and interventional filtering, subsequently selecting data that best activates these features. Tested on tasks like mathematical reasoning, summarization, and translation using the Gemma-2, LLaMA-3.1, and Qwen3 models, IGDS shows remarkable data efficiency. In the Math task, it outperforms full-dataset fine-tuning by 17.4% on Gemma-2-2B while utilizing considerably less data, effectively linking mechanistic interpretability with model optimization.