LOES: A Spectral Method for Layer-Wise Embedding Selection in Foundation Models

A recent study published on arXiv (2605.23033) questions the prevalent method of relying solely on the final layer or superficial combinations of foundation models for transfer learning. The researchers reveal that task-relevant information is unevenly distributed across different layers, making simple aggregation ineffective. Their geometric and empirical analyses across various modalities indicate that successful transfer learning hinges on pinpointing which layers contain task-discriminative features and understanding the geometric arrangement of their embeddings. They present Layer-wise Optimal Embedding Selection (LOES), a constructive spectral technique that identifies task-discriminative subspaces by reducing residual error while adhering to orthogonality and isotropy constraints. Additionally, they introduce Geometric Regularization Lo to align fine-tuning with this selection approach.