LLM Fairness Testing in Role-Playing Scenarios
A new empirical study from arXiv (2411.00585v2) investigates fairness in Large Language Models (LLMs) when they adopt specific roles. Researchers generated 550 social roles across 11 demographic attributes, producing 33,000 role-specific questions in Yes/No, multiple-choice, and open-ended formats to probe biases. The study aims to determine whether and to what extent social biases emerge during role-playing, a technique used to enhance LLMs' real-world utility. The work highlights the need for fairness testing in increasingly common role-playing applications.
Key facts
- Study conducted on arXiv preprint 2411.00585v2
- 550 social roles generated across 11 demographic attributes
- 33,000 role-specific questions created
- Question formats include Yes/No, multiple-choice, and open-ended
- Focus on fairness testing of LLMs in role-playing scenarios
- Role-playing is used to enhance LLM real-world utility
- Social biases in LLM outputs are a known issue
- Empirical study examines bias emergence during role-playing
Entities
Institutions
- arXiv