Equivariant Music Transformer: New AI Model for Music Recognition
A significant drawback has been discovered in conventional music transformers: they do not effectively recognize musical segments when altered in pitch or time, leading to unrelated representations for these changes. This deficiency in equivariance becomes more pronounced as models increase in size or undergo longer training, suggesting that extra capacity is devoted to memorizing specific patterns instead of understanding common musical frameworks. To tackle this issue, the research team introduces the Equivariant Music Transformer (EMT), which applies self-distillation to enforce equivariance by optimizing both next-token prediction and an additional equivariance regularization loss. This extra loss serves as a useful regularizer, enhancing next-token prediction and yielding equivariant latent representations. The research can be found on arXiv with the identifier 2608.03920.
Key facts
- Standard music transformers map time-shifted or pitch-transposed inputs onto uncorrelated representations.
- Equivariance in music transformers decreases as model size or training duration increases.
- The proposed Equivariant Music Transformer (EMT) enforces equivariance via self-distillation.
- EMT jointly optimizes next-token prediction and an auxiliary equivariance regularization loss.
- The equivariance loss improves next-token prediction and produces equivariant latent representations.
- The paper is announced as a cross-type on arXiv with identifier 2608.03920.
- The research highlights a trade-off between model capacity and equivariance in music AI.
- The findings suggest standard models allocate capacity to memorizing absolute patterns.
Entities
Institutions
- arXiv