Meta's Multi-Stage Sequence Model Achieves LLM-Style Scaling for Ads Ranking

Meta has unveiled a new multi-stage sequence model architecture designed for its advertising ranking systems, demonstrating predictable scaling laws akin to LLMs. This architecture separates offline user modeling from online ranking, facilitating the efficient scaling of transformer-based sequence models. By employing dense tokenization and target-aware attention, the model can directly learn feature interactions from data, minimizing the need for manual feature engineering. As a vital part of Meta's Generative Ads Recommendation Model (GEM), this platform has led to a 6% increase in conversions on Instagram, 3% on Facebook, and a 3.5% rise in Facebook ad clicks. The model asynchronously processes extensive user histories to create cached embeddings, while online ranking models utilize fresh signals for real-time scoring. Key scaling elements include balanced model shape, multi-stage tunability, sequence composition, and semantic feature representation. These insights are elaborated in the paper 'LLaTTE: Scaling Laws for Multi-Stage Sequence Modeling in Large-Scale Ads Recommendation'.