LLQR+SAM: Geometry-Aware Sharpness Minimization for Neural Networks

A novel optimization technique called LLQR+SAM has been developed by researchers, enhancing sharpness-aware minimization (SAM) through the integration of learned loss geometry. While SAM boosts generalization by adjusting parameters in high-curvature directions, it does so uniformly across all directions. LLQR+SAM merges SAM with a preconditioner derived from the LLQR framework, which reformulates steepest descent into a layerwise linear-quadratic regulator challenge. This preconditioner is updated gradually using a slow exponential moving average, providing a smoothed, low-resolution view of the loss landscape. Consequently, the SAM perturbation functions on this learned geometry at an accelerated pace. Theoretically, this dual-timescale approach strengthens SAM's ability to escape in directions that are locally sharp but flat in the average geometry, akin to maneuvering around potholes, thus enhancing generalization by adapting to the local curvature of the loss surface.