Bifurcation Theory Predicts Neural Network Concept Emergence in Real Time

A recent study published on arXiv (2605.24057) presents a bifurcation theory aimed at identifying when neural networks begin to form structured representations during their training, independent of labels. By examining a passive Gaussian mixture model probe linked to the developing encoder, the authors demonstrate that the emergence of structure aligns with a supercritical pitchfork bifurcation influenced by the loss Hessian. This results in a universal, label-free phase coordinate—the dynamic ratio β(t)/β_c(t)—which can be computed solely from hidden states. The theory has been tested in various scenarios, including sparse autoencoders on Pythia language models, self-supervised learning on CIFAR, and grokking in modular arithmetic, confirming four distinct transition regimes.