HYMELL: Hybrid Model Predicts LLM Inference Latency and Energy

A new framework named HYMELL (Hybrid Modeling for Energy and Latency of LLMs) has been developed by researchers to estimate the energy consumption and inference latency of large language models (LLMs). This three-tiered approach integrates analytical modeling with machine learning (ML) to tackle the increasing computational expenses, energy usage, and latency linked to LLM expansion. HYMELL functions through a hierarchy that includes the analytical assessment of basic operations, ML predictions for advanced components, and a comprehensive model that accounts for system-level overheads during both prefill and decode stages. It accommodates various architectures, such as dense and mixture-of-experts (MoE) feed-forward networks (FFNs), along with multi-head attention (MHA) and grouped-query attention (GQA) methods. Evaluated on an unspecified benchmark, the framework demonstrated its capability in accurately predicting latency and energy. This research is crucial for sustainable AI implementation and hardware-aware design, as precise estimations are vital for optimizing resource management and minimizing environmental effects. The paper can be found on arXiv with the identifier 2608.06723.