LLM-Simulated Examinees Align with Human Data via Cognitive Diagnostic Profiling
A novel zero-shot approach known as Cognitive Diagnostic Profiling (CDP) enhances the correspondence between LLM-simulated test-takers and actual human response data for psychometric calibration. Researchers examined eight LLM configurations using the Tatsuoka fraction-subtraction dataset, which includes 536 examinees, 15 items, and five attributes, under three conditions: no-profile, uninformative-CDP, and informative-CDP. CDP improved alignment across ability distribution, mastery profiles, and item difficulty, mitigating the excessive accuracy and uniformity often seen in LLM simulations. This technique encourages LLMs to create varied cognitive profiles based on binary mastery patterns derived from either uninformative or informative distributions, providing a budget-friendly alternative to expensive human data collection for initial test calibration.
Key facts
- Cognitive Diagnostic Profiling (CDP) is a zero-shot framework for psychometric calibration.
- CDP prompts LLMs to simulate examinees with diverse cognitive profiles.
- The Tatsuoka fraction-subtraction dataset includes 536 examinees, 15 items, and five attributes.
- Eight LLM configurations were evaluated under three conditions.
- CDP improved alignment at ability-distribution, mastery-profile, and item-difficulty levels.
- LLM-simulated responses are typically too accurate and uniform.
- CDP uses binary attribute-mastery patterns rendered as natural-language profiles.
- Profiles are sampled under uninformative or informative distributions.
Entities
Institutions
- arXiv