LLM Generalization Across Difficulty Levels: A Systematic Study

A recent paper on arXiv (2511.21692) explores the generalization abilities of large language models (LLMs) concerning varying task difficulties, a vital aspect for data evaluation and curation. The researchers highlight inconsistent findings in previous studies regarding whether training on simpler or more complex data results in superior outcomes, and whether improvements manifest in easier or more challenging test data. To investigate this, they perform a comprehensive assessment across multiple models, datasets, and specific difficulty categories. Utilizing outputs from thousands of LLMs and Item Response Theory (IRT), they rank examples in six datasets. Their difficulty assessments are based exclusively on the performance of various LLMs, omitting human judgment. This objective analysis reveals that generalization across difficulty levels is frequently constrained. The implications of their findings affect how training data is chosen and how model efficacy is measured, indicating that improvements may not transfer as anticipated across different difficulty tiers.