Research warns of analytic flexibility risks in using LLMs as human data substitutes

Social scientists increasingly employ large language models to generate synthetic datasets known as silicon samples, intended to replace human respondents in research. A new study examines how numerous analytic decisions impact the correspondence between these artificial samples and actual human data. Across two investigations, the research evaluated 252 distinct silicon-sample configurations for a controlled case study involving two social-psychological scales. These configurations assessed whether synthetic data could accurately recover participant rankings, response distributions, and correlations between scales. Substantial variation was found across all three evaluation criteria, with configurations performing well on one dimension often failing on others. The analysis was extended to a published real-world application of silicon samples. Key choices influencing outcomes include model selection, sampling parameters, prompt formatting, and the inclusion of demographic or contextual details. This work highlights methodological challenges in using AI to simulate human responses for social science research.