LENS: Evaluating Narrative Unlearning in LLMs

A novel assessment framework known as Level-based Evaluation of Narrative Suppression (LENS) has been developed to determine if machine-unlearning techniques can stop large language models from generating narratives aligned with disinformation. LENS evaluates the reproduction of target narratives on four levels: direct, attributed, contrastive, and abstract resistance. The research examines two narratives: one portraying Russia's invasion of Ukraine as a result of NATO's expansion, and another depicting the United States as either exploiting or neglecting Taiwan. The experiments involve four multilingual instruction models nearing 12 billion parameters: Lapa LLM, Gemma-12B, Qwen-14B, and TAIDE-Gemma. Additionally, the Suppression-Collapse Efficiency (SCE) score is introduced to serve as a metric for selecting checkpoints that encourage target-narrative suppression while discouraging poor outputs.