SaFE-Scale Framework Measures Clinical LLM Safety Across Scaling Conditions

A recent study has unveiled SaFE-Scale, a new framework designed to assess safety in clinical large language models (LLMs) by considering factors such as model scale, quality of evidence, retrieval methods, context exposure, and computational requirements during inference. The researchers emphasize that merely increasing accuracy does not ensure safer medical outcomes, as a few critical errors can significantly overshadow overall performance. To apply this framework, they introduced RadSaFE-200, a benchmark consisting of 200 multiple-choice questions focused on Radiology safety, featuring clinician-defined clean and conflict evidence, along with labels identifying high-risk errors, unsafe responses, and evidence contradictions. The evaluation included 34 locally deployed LLMs across six different deployment scenarios, such as closed-book prompting (zero-shot) and clean evidence. This research is available on arXiv with the identifier 2605.04039.