LLM-Based Difficulty Calibration for Programming Exams: A Multi-Evidence Study

A recent investigation published on arXiv (2608.07523) explores the application of large language models (LLMs) as supportive instruments for assessing the difficulty of programming exam questions across parallel classes, with the goal of enhancing fairness in evaluations. This research shifts the role of LLMs from simple benchmark targets to valuable sources of evidence that augment traditional assessment metrics. By integrating AI-generated insights with aggregated student results, item exposure, online-judge process data, and teacher evaluations, the study conducted an experiment where ten models tackled an eight-question final exam alongside 120 students. The correlation between the AI pass rate and student pass rate was positive (Spearman rho = 0.866, exact p = 0.0119), while a composite difficulty index based on solutions showed a negative correlation (rho = -0.905, exact p = 0.0046). The analysis included 79 problems from 11 parallel-class exams, indicating that LLMs could significantly assist educators in creating fairer assessments by combining AI evidence with human and process data for thorough evaluations.