LLMs Misjudge Difficulty of Fraction Problems
A study comparing LLM-generated difficulty ratings with empirical student performance on basic arithmetic reveals systematic misalignment for fraction items. Researchers had four LLM systems rate 32 arithmetic items on a 1-100 scale (N=640 ratings) and compared these to data from 770 Indonesian undergraduates using Classical Test Theory and Item Response Theory. While rank correlations were moderate (Spearman's rho 0.52-0.70), several fraction items rated as easy by LLMs were among the most difficult for students, indicating LLMs underestimate misconception-driven difficulty.
Key facts
- Study compares LLM difficulty ratings to empirical student performance on 32 arithmetic items
- Four LLM-based systems generated 640 difficulty ratings on a 1-100 scale
- Empirical data from 770 Indonesian undergraduates using CTT and 2PL IRT
- Rank correlations: Spearman's rho 0.52-0.70
- Systematic misalignment found in fraction items
- LLMs rated some fraction items as easy that were actually difficult for students
- Research published on arXiv (2607.26067)
- Title: 'The Easy Trap: Why LLMs Underestimate Misconception-Driven Difficulty'
Entities
Institutions
- arXiv
Locations
- Indonesia