No Gaussian Required: Contrastive Inverse Dynamics for JEPA World Models
A recent preprint on arXiv (2608.17542) introduces Action-Contrastive Masked Transition Modeling (AC-MTM), offering a fresh approach to the typical anti-collapse strategies found in Joint-Embedding Predictive Architectures (JEPAs). LeWorldModel (LeWM) utilizes SIGReg to avert encoder collapse by ensuring the latent distribution aligns with an isotropic Gaussian. The authors suggest that transition data itself can provide the necessary anti-collapse pressure. AC-MTM retains LeWM's forward latent-prediction goal while incorporating a training-only inverse-dynamics head that employs Action-NCE. This component requires each latent transition to pinpoint the corresponding action from others in the batch, a challenge unattainable for a collapsed encoder. The study cites previous works by LeCun (2022), Assran et al. (2023), and Bardes et al. (2022, 2024).
Key facts
- Paper arXiv:2608.17542 introduces Action-Contrastive Masked Transition Modeling (AC-MTM)
- AC-MTM is designed for Joint-Embedding Predictive Architectures (JEPAs)
- LeWorldModel (LeWM) uses SIGReg to prevent encoder collapse by matching an isotropic Gaussian
- AC-MTM replaces Gaussian regularization with contrastive inverse dynamics
- The method includes a training-only inverse-dynamics head trained with Action-NCE
- Each latent transition must identify the action that produced it among other actions
- A collapsed encoder cannot pass the discrimination task
- References prior work by LeCun, Assran, and Bardes
Entities
Artists
- Yann LeCun
- Assran
- Bardes
Institutions
- arXiv