Fast Entropic Approximations for SE and KL Divergence
A recent study published on arXiv introduces Fast Entropic Approximations (FEA), which are non-singular rational approximations for Shannon entropy (SE) and symmetrized Kullback-Leibler divergence (KL). These new approximations maintain essential mathematical characteristics and achieve mean absolute errors of approximately 10^-3, outperforming existing computational methods by 10-20 times. FEA facilitates computations that are up to twice as fast for SE and 37 times quicker for symmetrized KL. This research tackles the issue of gradient singularity near zero, which is a significant factor contributing to high costs, low robustness, and slow convergence in various fields, including physics, information theory, machine learning, and quantum computing.
Key facts
- Fast Entropic Approximations (FEA) are proposed for Shannon entropy and symmetrized Kullback-Leibler divergence.
- FEA are non-singular rational approximations that preserve main mathematical properties.
- Mean absolute errors are around 10^-3, 10-20 times better than comparable state-of-the-art approximations.
- FEA allows up to 2 times faster computation of Shannon entropy.
- FEA allows up to 37 times faster computation of symmetrized KL divergence.
- The paper addresses gradient singularity near zero in entropic measures.
- Applications include physics, information theory, machine learning, and quantum computing.
- The paper is available on arXiv with ID 2505.14234.
Entities
Institutions
- arXiv