DASH: Fast Differentiable Search for Hybrid Attention in LLMs

DASH, a novel and rapid differentiable search framework, has been developed for designing hybrid attention architectures in large language models (LLMs). These hybrid architectures, which integrate various attention mechanisms, are crucial for enhancing inference efficiency without compromising model quality. Current approaches often depend on manual guidelines or proxy selectors, while recent NAS-style methods like Jet-Nemotron necessitate 200 billion tokens for their search processes, rendering them impractical for everyday application. DASH transforms the discrete placement of attention operators into continuous architecture logits, generates reusable teacher-aligned linear candidates, and conducts architecture-only searches using fixed model and operator weights, thereby significantly lowering computational expenses. This framework allows for hybrid architecture searches within minutes on a single GPU, promoting broader adoption and addressing a key challenge in LLM design by automating the distribution of attention operators across layers.