RefusalBench: Benchmark Reveals LLM Refusal Rate Disparities on Biological Research Prompts

RefusalBench is a new standard that evaluates how well large language models handle biological research questions with varying levels of risk. It includes 141 prompts divided into 47 matched sets, ensuring a consistent approach while changing the risk categories (benign, borderline, dual-use). A special section with 15 prompts for positive control helps establish a baseline; interestingly, three models didn't refuse any of these. In a review from May 2026, 19 advanced models showed a wide range of refusal rates, from just 0.1% to 94.6% for identical prompts. While location didn't affect these rates, the model provider did, with Anthropic's API showing a high refusal prediction odds ratio of 21.03. This highlights the need for standardized evaluations.