Symmetric Attention Decomposition Balances Fidelity and Diversity in Diffusion Models
Researchers characterize the pre-softmax attention matrix in transformers as an associative memory matrix. By decomposing it into symmetric and skew-symmetric parts, they interpret the symmetric component as governing energy landscape structure and the skew-symmetric component as driving circulation. They derive Hopfield-style stability measures from the symmetric component, finding correlations with fidelity-diversity trade-offs in generation. A controllable knob is proposed to modulate this trade-off by altering circulation dynamics. Code is available on GitHub.
Key facts
- Pre-softmax attention matrix QK^T is characterized as an associative memory matrix encoding pairwise associations.
- Matrix is decomposed into symmetric and skew-symmetric parts.
- Symmetric component governs energy landscape structure; skew-symmetric drives circulation.
- Hopfield-style stability measures are derived from the symmetric component.
- Stability measures correlate with fidelity-diversity trade-offs in generation.
- A controllable knob modulates the trade-off by modifying circulation dynamics.
- Code is available on GitHub.
Entities
Institutions
- arXiv
- GitHub