LLM Safety Benchmarks: Code Quality and Community Adoption Study

A comprehensive evaluation of 31 LLM safety benchmarks—focused on prompt injection, jailbreak, and hallucination—alongside 382 non-benchmark papers as a control group, uncovers notable shortcomings in code quality and functionality. This research integrates automated static analysis, over 220 hours of human runnability testing, and bibliometric analysis. Results indicate that only 39% of benchmark repositories operate without changes, while 16% offer error-free installation instructions, and just 6% consider ethical implications despite containing potentially dangerous content. This study aims to fill the void in the systematic evaluation of benchmark code quality and the elements influencing community acceptance.