CCPO: Counterfactual Credit Assignment for Multi-Agent LLM Collaboration

A novel approach known as Collaborative Credit Policy Optimization (CCPO) tackles the challenge of credit assignment in multi-agent large language model (LLM) systems. This method transforms outcomes at the team level into individual learning signals for agents through two allocators: counterfactual credit estimation, which assesses an agent's contribution by contrasting actual results with hypothetical scenarios where the agent is absent, and verifier-anchored LLM self-evaluation, which employs constrained self- and peer-assessments to allocate credit while prioritizing the external verifier's outcome. The role-specific rewards generated can be applied with GRPO-style updates or other policy gradient techniques. This optimizer-agnostic strategy seeks to reduce free-riding in collaborative multi-agent LLM environments.