MIRA Benchmark Reveals LLMs Dilute Health Info for Low-Literacy Users
The Medical Information Response Audit (MIRA) was developed by researchers as a bilingual standard to evaluate whether large language models (LLMs) deliver consistent medical information based on varying user phrasing. Comprising 4,320 prompts derived from 60 medically reviewed, low-risk health inquiries, MIRA was used to test five popular LLMs. The findings indicated that, although all models addressed the medical questions, responses to signals of low health literacy frequently lacked essential information, provided fewer actionable steps, and offered diminished support for independent decision-making. This phenomenon has been labeled Differential Information Dilution (DID). The impact of language was found to be specific to each model rather than uniformly detrimental. The study is available on arXiv (ID: 2605.28025).
Key facts
- MIRA is a bilingual benchmark for medical information response audit.
- It contains 4,320 prompts from 60 medically reviewed health questions.
- Five mainstream LLMs were tested.
- Models answered all medical questions.
- Low health-literacy responses omitted more key information.
- Low health-literacy responses provided fewer concrete next steps.
- Low health-literacy responses offered less support for independent judgment.
- The pattern is called Differential Information Dilution (DID).
- Language effects were model-specific.
- The study is published on arXiv with ID 2605.28025.
Entities
Institutions
- arXiv