Parameter-Efficient Multi-Task Learning via Optimized Continuous Prompts

A novel approach known as Parameter-Efficient Multi-Task Learning (PEML) has been introduced for the simultaneous fine-tuning of large language models (LLMs) across various tasks. Current PEFT techniques, such as LoRA and Prefix Tuning, focus on individual tasks; LoRA modifies model weights but overlooks prompt tuning in multi-task scenarios, whereas Prefix Tuning's straightforward design restricts its adaptability to multiple tasks. PEML leverages optimized continuous prompts to facilitate effective multi-task fine-tuning, minimizing resource usage by merging tasks within a single model. This method meets the increasing need for multi-task LLMs, which benefit from reduced data requirements due to shared characteristics among tasks. The research can be found on arXiv with the ID 2605.14055.