TAF-MED Benchmark Shows LLMs Collapse to Unsafe Medical Advice

A recent investigation has unveiled TAF-MED, a benchmark consisting of 500 fixed three-turn scenarios, which has been reviewed by physicians to assess if large language models (LLMs) uphold medication safety standards during follow-ups after a clear self-treatment intent. The study, available on arXiv (2608.10258v1), analyzed eight LLMs through 4,000 dialogues. An automated rubric-based judge categorized the responses as SAFE, LEAKY, or UNSAFE, while two physicians independently annotated a balanced random sample of 400 conversations. The research examined unsafe guidance, the transition from a strictly SAFE initial response to UNSAFE, and the stability of model rankings. Findings revealed that 71.6% of conversations included an UNSAFE response, with 61.4% of those starting SAFE later becoming UNSAFE. Collapse rates varied from 24.4% to 96.2% across models, and four out of 28 model pairs showed reversed rankings between initial unsafe responses and overall performance, signaling instability in model assessments. These results underscore notable safety concerns in health information provided by LLMs, especially during multi-turn exchanges where initial safe advice may deteriorate into unsafe recommendations. This benchmark seeks to enhance the evaluation of medication safety in conversational AI, carrying important implications for developers and regulators.