Meta Engineers Detail System That Converts Idle AI Inference Capacity Into Better Ads Recommendations

Meta has unveiled a system designed to optimize unused inference capacity throughout its network, thereby improving ad recommendations without needing extra resources. This initiative tackles the "inference capacity paradox," where global limitations exist alongside local surplus due to discrepancies in provisioning, demand, and available resources. The company utilizes two elasticity strategies: Rank More Ads (RMA), which boosts candidate evaluations by leveraging CPU and accelerator excess, and Serve Better Models (SBM), which directs requests to superior models using only accelerator surplus. Both RMA and SBM are safeguarded by physical and logical isolation. A central controller oversees candidate volume and target coverage, while load shedding adapts to changes in capacity. Meta plans to refine these strategies and automate the allocation of inference capacity.