DEI Prompts Cause AI Models to Fabricate Patient Demographics

A new study from arXiv (2608.15254) reveals a significant side effect of prompting language models with diversity, equity, and inclusion (DEI) guidelines in clinical settings. The research, conducted across 47 models, four medical benchmarks, and 376,000 responses, found that appending a single DEI prompt to a medical question causes models to inject demographic attributes—such as race, socioeconomic status, or sex—that were not present in the original question. This phenomenon, termed 'demographic injection,' rewrites the patient's identity and occurs in all 47 models tested. The injection rate jumps from 0.7% to 33.1% (a 47-fold increase) when a DEI prompt is added, and this effect is attributed to the equity content itself rather than the added length, as a length-matched control shows an 18-fold increase (p=1.4x10^-14). While most injected content is a general population statement that does not alter the answer, a smaller subset attaches attributes to the specific patient or changes the selected option (0.25-2.4%). The study highlights a potential risk in following clinical-AI guidance that recommends DEI-aware prompting, as it may inadvertently misrepresent patients and affect diagnostic accuracy. The findings underscore the need for careful evaluation of prompt engineering in medical AI applications.