HoReN: A New Method for Sequential Model Editing in LLMs
Researchers have introduced HoReN, an innovative parameter-preserving editor based on a codebook for large language models, aimed at resolving the issue of refreshing outdated or incorrect factual information without the need for retraining. HoReN encapsulates a single MLP layer within a discrete key-value codebook, where each entry functions as both a knowledge-memory key and a contemporary Hopfield stored pattern. This strategy improves routing and mitigates performance decline at scale, addressing the shortcomings of current locate-then-edit methods that disrupt retained knowledge, as well as external memory techniques that encounter routing difficulties. Designed for continuous model editing, this method facilitates specific behavior modifications while preserving the overall integrity of the model. The research is available on arXiv under ID 2605.08143.
Key facts
- HoReN is a codebook-based parameter-preserving editor for large language models
- It wraps a single MLP layer with a discrete key-value codebook
- Each codebook entry is interpreted as both a knowledge-memory key and a modern Hopfield stored pattern
- The method addresses performance degradation at scale in model editing
- It overcomes limitations of locate-then-edit and external memory approaches
- The paper is available on arXiv under ID 2605.08143
- HoReN is designed for lifelong model editing
- It enables targeted behavior updates without retraining
Entities
Institutions
- arXiv