SCRuB: A Framework for Evaluating LLM Reasoning on Social Concepts
A new framework called SCRuB (Social Concept Reasoning under Rubric-Based Evaluation) has been developed by researchers to evaluate how Large Language Models (LLMs) interpret abstract social concepts, including norms, culture, and institutions. While existing studies on LLM reasoning primarily concentrate on mathematical or technical challenges, the area of social concept reasoning has not received adequate attention, despite its significance for models functioning as social agents. SCRuB employs a three-phase pipeline to tackle task indeterminacy: it begins with prompt creation from reliable sources, followed by response generation from both human experts and models, and concludes with a comparative assessment using a five-dimensional critical thinking rubric. To facilitate generalization, the framework includes a Panel of Disciplinary Perspectives ensemble, which has been validated against expert evaluations. This research is available as arXiv preprint 2605.06444.
Key facts
- SCRuB stands for Social Concept Reasoning under Rubric-Based Evaluation
- The framework targets LLM reasoning about social concepts like norms, culture, and institutions
- Most existing LLM reasoning studies focus on mathematical or technical tasks
- SCRuB uses a three-phase pipeline: prompt construction, response generation, and comparative evaluation
- Evaluation uses a five-dimensional critical thinking rubric
- A Panel of Disciplinary Perspectives ensemble is used for generalization
- The work is published as arXiv:2605.06444
- The framework is designed for task indeterminacy in social concept reasoning
Entities
Institutions
- arXiv