LLMs Fail as Synthetic Users in Survey Simulation Benchmark

A new study from arXiv (2607.26348) systematically evaluates whether large language models (LLMs) can replace human respondents in survey-based research. The researchers tested four models from two families (ranging from 8B to frontier capability) on two independent datasets: the U.S. General Social Survey and the World Values Survey. Under demographic prompting and standardized simulation protocols, no LLM outperformed the strongest non-LLM baseline fitted on held-out human data. At the individual level, all models failed to replicate human response patterns. The failures were consistent across both domains, all four models, and both model families. The study provides an evaluation framework for synthetic-user systems and highlights the limitations of current LLMs in simulating human survey responses.