Energy Benchmark of Locally Deployed LLMs on Consumer GPU
There's a new preprint on arXiv (2608.00008) that presents a reproducible energy benchmark for nine open-source large language models (LLMs), each with parameters ranging from 1 billion to 7 billion. The tests were conducted on a single consumer GPU, the RTX 4060Ti with 16GB. The study focuses on the often overlooked energy costs of running LLMs on-site, using the Ollama inference engine and monitoring GPU power at a rate of 2Hz via nvidia-smi. Key metrics assessed include average and peak power, energy used per prompt (in joules), energy per output token, and throughput. Findings reveal that energy efficiency varies based on model design and quantization, with gemma3:1b and llama3.2:1b having the lowest energy costs and highest throughput. This research aims to fill a gap in LLM assessments that typically emphasize accuracy over energy consumption, providing a framework for future studies.
Key facts
- Preprint arXiv:2608.00008 announces new research on energy efficiency of locally deployed LLMs.
- Benchmark covers nine open-source LLMs with 1B to 7B parameters.
- Hardware used: single consumer GPU (RTX 4060Ti 16GB).
- Inference engine: Ollama; power draw sampled at 2Hz via nvidia-smi.
- Metrics: mean/peak power, J/prompt, J/token, tok/s.
- gemma3:1b and llama3.2:1b had lowest energy cost (0.56 and 0.65 J/token) and highest throughput (>170 tok/s).
- Energy efficiency driven by model architecture and quantization, not just parameter count.
- Study addresses lack of energy benchmarks for on-premise LLM inference.
Entities
Institutions
- arXiv
- Ollama