Cross-Modal Skill Injection for VLMs: A Systematic Study

A recent paper on arXiv (2605.19523) thoroughly examines cross-modal skill injection, a technique designed to transfer specialized knowledge from Large Language Models (LLMs) to Vision-Language Models (VLMs) without needing extra training data or heavy computational demands. Unlike traditional methods that combine similar LLMs by pooling their abilities, this approach seeks to create new cross-modal skills by incorporating a domain-expert LLM into a VLM. The research investigates various scenarios, techniques, and hyperparameters to tackle the issue of VLMs' difficulty in adapting to rapidly changing domain-specific skills. Traditional methods like Supervised Fine-Tuning (SFT) require large datasets and significant computational power, making model merging a more efficient solution. This paper addresses a gap in the literature regarding the systematic analysis of cross-modal skill injection's applicability and methods.