JetViT: Efficient High-Resolution Vision Transformer with Post-Training Attention Search

A group of researchers has unveiled JetViT, a series of hybrid-architecture Vision Transformer (ViT) models that achieve the same accuracy as leading full-attention vision foundation models while significantly enhancing inference efficiency for high-resolution images. The primary breakthrough is the Post-Training Attention Search, an acceleration framework that transforms pre-trained full-attention ViTs into efficient hybrid-attention models by identifying and substituting unnecessary full-attention blocks with linear or window-attention alternatives. This framework retains MLP and attention weights from the original model and navigates the design space through three essential phases: optimizing linear-attention block design, determining the optimal mix of linear and window-attention blocks, and recognizing and maintaining vital full-attention blocks. JetViT shows substantial efficiency improvements in high-resolution image tasks without compromising accuracy. The research is available on arXiv under ID 2605.26636.