Meta-Soft: Dynamic KV Cache Compression for LLMs

Meta-Soft is a new framework for compressing the key-value (KV) cache in large language models (LLMs), addressing memory blow-up and reduced decoding efficiency during long-context processing. Unlike existing methods like Judge Q, which use fixed soft tokens and static parameters, Meta-Soft employs a dynamic compression approach based on probe-driven context integration. It builds a meta-library with a learnable orthogonal basis matrix and uses a selector network with Gumbel-Softmax to produce differentiable sparse combinations. This allows the model to adapt to different input prompts and precisely capture task relevance, preventing irreversible information loss from permanent eviction of KV pairs.