Predictive Coding Networks Match Backpropagation in Infinite Limits
A recent research paper from arXiv (2602.07697v2) reveals that predictive coding (PC) networks, which serve as a biologically feasible substitute for backpropagation (BP), can yield equivalent gradient calculations under certain circumstances. For linear residual networks, the parameterizations that maintain stability in both width and depth are consistent for both PC and BP. When the model's width significantly surpasses its depth, the energy of PC, with balanced activities, aligns with the quadratic loss of BP. Experimental results indicate that achieving activity equilibrium enables PC to compute gradients identical to those of BP, thereby addressing issues related to scalability and theoretical discrepancies.
Key facts
- Predictive coding is a biologically plausible alternative to backpropagation.
- PC minimizes an energy function with respect to network activities before updating weights.
- Recent work improved training stability of deep PC networks using BP-inspired reparameterizations.
- The study examines infinite width and depth limits of PCNs.
- For linear residual networks, width- and depth-stable parameterizations are identical for PC and BP.
- PC energy converges to quadratic BP loss when width is much larger than depth.
- Experiments show PC computes same gradients as BP when activity equilibrium is reached.
- The paper is available on arXiv with ID 2602.07697.
Entities
Institutions
- arXiv