LLM Agents for Survey Prediction: Demographics vs. In-Domain Data
An arXiv study (2605.16303) evaluates two methods for employing LLM agents to forecast survey answers: one relies solely on demographics (such as age, gender, and income), while the other incorporates a broader range of in-domain survey responses. This research is conducted using the Survey of Health, Ageing and Retirement in Europe (SHARE) and concentrates on five financial variables. Findings indicate that agents based only on demographics display a central tendency bias and an unrealistic level of accuracy, as they do not replicate the incorrect or 'don't know' responses that are common among human participants.
Key facts
- arXiv paper 2605.16303 compares LLM agents for survey prediction.
- Demographics-only agents use country, age, gender, employment, income, education, marital status.
- Survey agents use a larger set of in-domain responses from SHARE.
- SHARE is a multidisciplinary, cross-national survey on health, ageing, and retirement in Europe.
- Five variables from three policy-relevant constructs around personal finance were tested.
- Demographics-only agents exhibited central tendency bias toward population means.
- Demographics-only agents were unrealistically accurate, missing incorrect and 'don't know' responses.
- The study highlights limitations of demographics-only LLM agents for survey modeling.
Entities
Institutions
- arXiv
- Survey of Health, Ageing and Retirement in Europe (SHARE)
Locations
- Europe