SymbolicLight V1: Spike-Gated Language Model Achieves High Sparsity
SymbolicLight V1 has been unveiled by researchers, featuring a spike-gated dual-path language model that integrates binary Leaky Integrate-and-Fire spike dynamics alongside a continuous residual stream. The model's innovative Dual-Path SparseTCAM module substitutes traditional dense self-attention with an exponential-decay aggregation for long-range memory and a spike-gated local attention mechanism for short-range accuracy. A version with 194M parameters, trained from the ground up on a 3B-token Chinese-English dataset, achieved a validation perplexity of 8.88-8.93 over four separate trials, exhibiting more than 89% activation sparsity. While it falls short of GPT-2 201M by 7.7% in perplexity, it outperforms GPT-2 124M. The architecture features a context-conditioned decoding head and a bilingual tokenizer. This research was published on arXiv.
Key facts
- SymbolicLight V1 combines binary LIF spike dynamics with a continuous residual stream.
- Dual-Path SparseTCAM module replaces dense self-attention.
- 194M-parameter model trained on 3B-token Chinese-English corpus.
- Validation perplexity of 8.88-8.93 across four runs.
- Over 89% per-element activation sparsity achieved.
- Trails GPT-2 201M by 7.7% in PPL.
- Surpasses GPT-2 124M in PPL.
- Published on arXiv with ID 2605.21333.
Entities
Institutions
- arXiv