Local LLMs Achieve 43-45% Accuracy in Python Bug Detection

An empirical investigation was carried out to evaluate the effectiveness of locally implemented large language models (LLMs) for detecting Python bugs in real-world applications, utilizing the BugsInPy benchmark. The research focused on LLaMA 3.2 and Mistral, analyzing 349 bugs from 17 different projects through a zero-shot prompting method at the function level, alongside an automated keyword-based evaluation framework. Findings indicate that models run locally achieve an accuracy ranging from 43% to 45%, often providing partially correct answers that highlight problematic code areas without offering specific solutions. The study underscores significant performance variations among projects and addresses limitations in previous research that depended on cloud-based models or specialized hardware, which restricts use in privacy-sensitive or resource-limited settings.