JEPA-DNA: A New Framework to Enhance Genomic Foundation Models

A recent study presents JEPA-DNA, a continual training framework that is model-agnostic, merging a Joint-Embedding Predictive Architecture (JEPA) with conventional generative objectives to enhance Genomic Foundation Models (GFMs). This paper, accessible on arXiv (arXiv:2602.17162), critiques existing GFMs, which generally depend on Masked Language Modeling (MLM) or Next-Token Prediction (NTP), for their emphasis on token-level reconstruction, neglecting broader functional context. JEPA-DNA improves this by supervising global sequence embeddings in a latent space, compelling models to predict functional representations of masked genomic segments, thus shifting the focus from token recovery to semantic alignment. Evaluated across 17 varied genomic benchmark tasks, it demonstrated notable improvements in linear probing and zero-shot performance, irrespective of the GFM architecture or generative objective. The authors assert that JEPA-DNA sets a new benchmark for genomic foundation models. This preprint has been announced with a type 'replace', signifying a revised version, and is pertinent to genomics, artificial intelligence, and machine learning, with implications for deciphering the 'Laws of Nature' encoded within genomic sequences.