Semantic Stability in Clinical LLMs: Evaluating Sensitivity to Prompt Variations

A new study from arXiv (2605.30646) investigates how Large Language Models (LLMs) respond to semantically equivalent but linguistically varied prompts in clinical settings. The researchers propose a semantic verification framework using Natural Language Inference (NLI) to filter meaning-preserving prompt variations, refined by an LLM-as-a-judge and audited by a clinical expert. They introduce three metrics: Meaning-Preserving Variation Sensitivity (MVS), confidence variation, and others to quantify model sensitivity. The work highlights risks in healthcare where subtle rephrasing can alter predictions, emphasizing the need for robust evaluation methods.