New AI Method Improves Survey Response Simulation Accuracy
A novel two-phase fine-tuning technique known as Distribution Shift Alignment (DSA) greatly enhances the ability of large language models to replicate human responses in surveys. Current zero-shot methods face challenges with prompt sensitivity and accuracy issues, while traditional fine-tuning largely conforms to training set distributions, often failing to surpass their accuracy levels. DSA overcomes these challenges by aligning output distributions and managing distribution shifts across various contexts. Instead of simply adapting to training data, DSA captures how these distributions evolve, allowing it to yield results that are significantly more aligned with the true distribution than the training data. This approach consistently surpasses others across five public survey datasets, presenting a more effective method for lowering the expenses of large-scale data collection via LLM simulation.
Key facts
- Distribution Shift Alignment (DSA) is a new two-stage fine-tuning method for LLMs
- DSA improves simulation of human survey responses by aligning output distributions and distribution shifts
- Existing zero-shot methods suffer from prompt sensitivity and low accuracy
- Conventional fine-tuning approaches mostly fit training set distributions and struggle to exceed training set accuracy
- DSA learns how distributions change rather than just fitting training data
- DSA produces results substantially closer to true distributions than training data
- The method consistently outperforms other approaches on five public survey datasets
- LLMs offer promising potential for reducing costs of large-scale data collection through survey simulation
Entities
—