ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Models

The newly introduced framework, ZeroUnlearn, tackles the issue of erasing sensitive data from large language models without the need for expensive retraining. This approach reconceptualizes machine unlearning as a knowledge re-mapping challenge through model editing. It employs a few-shot technique to replace sensitive information with a neutral target state, effectively discarding original representations. By utilizing a multiplicative parameter update with a closed-form solution, it ensures representational orthogonality, facilitating efficient and precise unlearning. Additionally, a gradient-based variant broadens the framework for multi-sample unlearning. Experiments indicate its effectiveness in maintaining model utility while removing specific knowledge. The full paper can be found on arXiv.