New Benchmark Estimates LLM Parameter Counts via Factual Knowledge

Researchers have developed a novel technique to estimate the parameter counts of closed-source large language models by evaluating their factual knowledge. They created Incompressible Knowledge Probes (IKPs), consisting of 1,400 factual questions categorized into 7 levels of obscurity, aimed at identifying knowledge that cannot be inferred through reasoning or improved by architectural changes. By establishing a log-linear relationship between IKP accuracy and parameter count using 89 open-weight models (ranging from 135M to 1,600B parameters) from 19 different vendors, they achieved an R² value of 0.917. The leave-one-out cross-validation revealed a median fold error of 1.59×, with 68.5% of estimates falling within 2× and 87.6% within 4×. This method leverages the fundamental requirement that storing F facts necessitates at least F/(bits per parameter) weights, offering a more precise estimate than inference economics, which incurs over 2× uncertainty due to hardware, batching, and serving-stack assumptions. The research can be found on arXiv under ID 2604.24827.