Certifying Non-Constant Latent Codes in Teacher-Guided VAEs
A recent theoretical finding has determined a precise threshold for identifying input-independent constant collapse within variational autoencoders (VAEs). Given a fixed nonconstant teacher distribution T(·|x), the optimal constant student corresponds to the dataset-average teacher distribution, with an alignment cost represented by the teacher mutual information I_T(X;T). This result offers a quantifiable certificate: if a strictly latent-only raw witness maintains an alignment loss below this threshold with a safety margin, it cannot be constant concerning the input. Experiments conducted on CIFAR-100, involving per-seed teacher searches, demonstrate that complete training remains on the certified side, whereas removing alignment pushes the witness into the constant regime.
Key facts
- Posterior collapse in VAEs is often diagnosed by symptoms like small KL term or weak latent code use.
- The paper studies a concrete failure mode: input-independent constant collapse.
- An exact threshold is derived for constant collapse in teacher-guided VAEs.
- The best constant student is the dataset-average teacher distribution.
- The alignment cost equals the teacher mutual information I_T(X;T).
- A latent-only raw witness with alignment loss below this threshold cannot be constant.
- Experiments on CIFAR-100 validate the theoretical certificate.
- Full training stays on the certified side of the boundary.
Entities
—