LLM-Metrics: Measuring Research Impact Through LLM Memory

A team of researchers has introduced LLM-Metrics, an innovative metric intended for evaluating the impact of research through the parametric memory of large language models (LLMs). Their hypothesis suggests that papers with greater influence garner more visibility within the academic sphere, subsequently contributing to LLM training data and enhancing parametric memory. To validate this concept, the researchers created four categories of multiple-choice probes focusing on title, author, method, and venue recognition. They assessed 549 computer science papers from 2023-2024 using 17 LLMs, which varied from 0.5B to 72B parameters across six different vendors. Out of the 17 models, 15 yielded positive predictions, with 9 showing significance at p < 0.05, demonstrating an overall Spearman correlation. This method seeks to overcome the shortcomings of citation counts, including temporal lag, disciplinary bias, and Matthew effects.