HOBA: Hierarchical RL Framework for Adaptive Online Advertising Bidding

A novel hierarchical reinforcement learning framework named HOBA (Hierarchical On-policy Bidding Agents) has been introduced to overcome the shortcomings of existing online advertising bidding systems. These systems typically rely on various offline-trained expert models such as PID controllers, model predictive control, and offline RL policies, but they struggle with adapting to fluctuating auction markets and often require expensive manual adjustments. HOBA separates strategic reasoning, model selection, and bid execution into three distinct time scales. At the top level, a large language model deduces hyperparameters from contextual signals through a Think-Act-Observe-Reflect loop utilizing past experiences. The mid-level employs a SARSA agent that adjusts model selection to remove bias, while the low level features a dynamic pool of experts, including PID, MPC, IQL, and Decision Transformer models, allowing for online adaptation without manual input.