Self-Supervised Encoders and the Information Bottleneck

A recent study published on arXiv introduces a framework for encoder-decoder learning that integrates geometric and information-theoretic concepts, grounded in the Information Bottleneck (IB) principle. By redefining IB as a rate-distortion challenge utilizing Kullback-Leibler divergence for distortion, the researchers demonstrate that optimal representations manifest as soft clusterings within the predictive manifold of the probability simplex, allowing for a linear decoder in its canonical form. They also derive transformations from flat Dirichlet distributions to exponential and isotropic Gaussian forms, linking maximum entropy priors to Euclidean space while quantifying entropy overhead. The proposed Sketched Isotropic Gaussian Regularization (SIGReg) offers a Gaussian relaxation of this concept, influencing rate accounting without affecting achievable predictions. The paper can be found on arXiv with ID 2604.27743.