Probabilistic Circuits' Curvature Decomposition and Adaptive Regularizer

A recent study published on arXiv (2608.12869) introduces a compositional curvature theory for Probabilistic Circuits (PCs), which are generative models enabling precise inference. Unlike deep neural networks, PCs facilitate exact calculations of the Hessian trace of the log-likelihood, reflecting the curvature of the loss surface. Previous research has applied global regularization to this trace to promote flatter optima; however, the authors argue that this method can be incorrectly specified for PCs due to their compositional nature. They demonstrate that the contribution of each sum node to the Hessian trace is a product of its circuit flow and a local sharpness term linked to its output distribution. This finding clarifies why global sharpness regularization tends to be depth-biased, potentially causing underfitting. Consequently, they propose an adaptive sharpness-aware regularizer that penalizes nodes based on their local curvature while maintaining closed-form tractability. The authors, researchers in the field, highlight the theoretical implications for the geometry of probabilistic circuits and suggest a practical enhancement for their training.