Residual Dominance Explains Last-Item Reliance in SASRec Recommenders
A recent paper on arXiv (2608.14021) explores the reasons behind the strong dependence of transformer-based sequential recommenders, like SASRec, on the latest interaction during inference. The researchers utilize prediction-time diagnostics alongside a norm-based examination of the complete attention block to reveal the underlying structural mechanism. Their findings indicate that models akin to SASRec show a pronounced reliance on the most recent item. Further investigation shows that while self-attention gathers contextual data, the residual connection significantly skews the overall representation towards contributions from the same position, a phenomenon they label 'residual dominance.' To test this theory, they apply residual scaling at inference time, which demonstrates a consistent trade-off between structural mixing and last-item dependence, while a decrease in residual strength recovers some final-position misses, enhancing understanding of causal self-attention recommenders and their potential for improved robustness.
Key facts
- Paper arXiv:2608.14021, announced as new.
- Focuses on transformer-based sequential recommenders with causal self-attention.
- SASRec-style models show highly localized last-item reliance.
- Residual connections cause 'residual dominance' in full-block representations.
- Inference-time residual scaling used as diagnostic intervention.
- Residual strength changes induce monotonic trade-off between mixing and last-item reliance.
- Reducing residual strength recovers some final-position misses.
- Provides structural account of last-item reliance in recommenders.
Entities
Institutions
- arXiv