AMMA: Memory-Centric Architecture for Low-Latency Long-Context LLM Attention

AMMA has unveiled a new design that focuses on memory by utilizing a multi-chiplet approach to cut down on latency in long-context attention for large language models (LLMs). Instead of relying on GPUs, which struggle with the heavy memory demands during the decode phase, AMMA integrates HBM-PNM cubes to improve performance. This change effectively doubles memory bandwidth, which is crucial for tasks reliant on memory. As context lengths grow to one million tokens in reasoning and agent applications, attention latency has become a major challenge for users. The new architecture aims to improve efficiency across LLM serving frameworks, including attention-FFN disaggregation and NVIDIA's Rubin GPU-LPU platform.