SymNoise Boosts LLaMA-2-7B to 69% on AlpacaEval
A new paper on arXiv (2605.23171) introduces SymNoise, a fine-tuning method that uses symmetric noise in embeddings to improve language model performance. The authors provide theoretical and empirical analysis showing that uniform, Gaussian, and symmetric noise types perform comparably, challenging NEFTune's claim that uniform noise is superior. When fine-tuning LLaMA-2-7B on Alpaca, SymNoise achieves 69.04% on AlpacaEval, compared to 29.79% with standard techniques and NEFTune's benchmark. The method regulates local curvature more stringently, enhancing model function.
Key facts
- Paper arXiv:2605.23171 introduces SymNoise
- SymNoise uses symmetric noise in embeddings
- LLaMA-2-7B fine-tuned on Alpaca achieves 69.04% on AlpacaEval
- Standard techniques yield 29.79% on AlpacaEval
- NEFTune previously set benchmarks with uniform noise
- Authors show uniform, Gaussian, and symmetric noise perform comparably
- Method regulates local curvature more stringently
- Paper provides theoretical and empirical analysis
Entities
Institutions
- arXiv