Study Warns LLMs Unsafe for Autonomous Clinical Decision Support
A recent study published on arXiv (2607.28677) asserts that large language models (LLMs) are not sufficiently safe for independent clinical decision-making, especially when it comes to triaging patients without clinician supervision. While the authors recognize that LLMs can pass medical licensing examinations and demonstrate diagnostic reasoning comparable to that of physicians in controlled scenarios, their focus is on the critical issue of autonomous triage for self-presenting patients. They argue that there is currently no evidence supporting the safety of LLMs for this purpose. The main concern lies not in medical knowledge gaps, but in the reliability of clinical assessments, as these models are not designed to prioritize rare yet critical diagnoses. The authors stress that successful triage involves complex decision-making, where missing a single critical diagnosis can have severe consequences. This Perspective piece highlights that the ability of LLMs to pass exams does not equate to ensuring safety in real-world clinical environments, urging caution in their application in high-stakes situations.
Key facts
- Paper on arXiv (2607.28677) warns LLMs are unsafe for autonomous clinical decision support.
- LLMs pass medical licensing exams and can rival physicians in diagnostic reasoning in curated cases.
- LLMs are being used for symptom assessment, clinical decision support, administrative documentation, and alert enhancement.
- The paper focuses on autonomous triage of undifferentiated patients with little or no clinician in the loop.
- Evidence of safety for autonomous triage does not yet exist.
- The gap is in fidelity of clinical evaluation, not medical knowledge.
- Models optimized for text continuation are not optimized for safe action when the safe answer is the improbable diagnosis.
- Safe triage is a sequential decision under asymmetric cost, where a single catastrophic miss outweighs many correct diagnoses.
Entities
Institutions
- arXiv