LLM Social Simulators Need Reason Audits, Not Just Outcome Checks

A recent preprint on arXiv (2607.24649) suggests that the assessment of large language models functioning as social simulators—like synthetic survey participants—is inadequate. The researchers argue that while aligning with human outcomes (such as final responses) is essential, it is insufficient, as a simulator may arrive at the correct answer using flawed reasoning. To tackle this issue, they propose a technique utilizing signed reason states Z, where positive signs endorse adoption and negative signs oppose it. In a study involving 94 participants assessing three sunscreen concepts, respondents provided open-ended rationales. The findings indicate that using human rationale-derived reasons significantly enhances the prediction of outcomes, highlighting that reason-based audits can expose discrepancies between human and simulated reasoning.