Feature Learning Equation Links Weight Gram Matrix to Neural Network Dynamics

A recent publication on arXiv presents a framework focused on features for the analysis of training in deep neural networks. The authors put forth the Feature Learning Equation, which highlights the weight Gram matrix as crucial for understanding feature dynamics. This perspective enables the interpretation of gradient descent as a process that induces a theoretical evolution of features, with its covariance structure—referred to as Virtual Covariance—describing how representations change throughout training. Additionally, they introduce Target Linearity, a metric for assessing the linear relationship between features and targets. Their examination of training and layer-wise dynamics reveals that deep networks progressively adjust representations towards a target-linear configuration. The paper can be found at arXiv:2605.06258.