Multi-Agent RL Method Retains Suboptimal Actions for Shifting Optima

A novel approach for cooperative multi-agent reinforcement learning (MARL), called Successive Sub-value Q-learning (S2Q), has been introduced by researchers. Unlike traditional value decomposition methods that focus on a single best action, S2Q develops several sub-value functions, allowing it to maintain access to alternative high-value actions. This flexibility helps the algorithm adjust to shifts in the underlying value function during training, preventing it from settling on suboptimal policies. By integrating these sub-value functions into a Softmax-based behavior policy, S2Q promotes continuous exploration and quick adaptation to changing optima. Tests on demanding MARL benchmarks demonstrate that S2Q consistently surpasses a range of MARL algorithms in terms of adaptability and performance. The code is available to the public.