Reduced Matrix Multiplication: Input-Adaptive Method for Efficient LLM Inference

A recent study published on arXiv (2608.13426) presents Reduced Matrix Multiplication (RMM), an inference technique that adapts to input without requiring training, aimed at decreasing the computational demands of Transformer-based language models. This approach selects key slices along the contraction dimensions of matrix multiplications without altering model weights, allowing for a flexible balance between accuracy and efficiency based on a retention ratio. Tests conducted on models with parameters ranging from 1B to 70B indicate that the tolerance for reduction differs by model type, task, component, and retention ratio, often improving with larger models. RMM proves effective even with moderate reductions in various settings, including discriminative, autoregressive generation, and long-context scenarios. The principles also apply to multimodal vision-language inference, with mechanistic ablations provided to clarify the method's functionality.