Symmetry Transfer in LLMs via Layer-Peeled Optimization

A new study analyzes whether pretraining large language models by minimizing cross-entropy loss for next-token prediction induces geometric structure in learned weights and context embeddings. Using a constrained layer-peeled optimization program as a tractable surrogate, the authors prove that symmetries in target next-token distributions transfer to global minimizers in a group-theoretic sense. Specifically, when target tokens exhibit cyclic-shift symmetry (e.g., days of the week, months of the year), the optimal logit matrix becomes exactly circulant, and Gram matrices of context embeddings reflect the same symmetry. The work provides mathematical foundations for understanding how optimization shapes representations in LLMs.