Quadratic Model Predicts LLM Optimization Dynamics

A new study tests the quadratic model as a simple yet predictive framework for understanding neural network optimization in large language models (LLMs). The research, published on arXiv (2607.21716), stress-tests this idealized model on a 150M-parameter LLM trained on 3 billion tokens. Findings show that Taylor expansions of the model and loss function at intermediate checkpoints can accurately forecast optimization dynamics over windows covering up to 10% of training. The authors then analyze the Hessian spectrum and local stability using Lanczos quadrature with deep probes. This work suggests that even the simplest optimization models can be surprisingly effective in complex LLM settings.