AI Feedback in Chess Reveals Self-Selection Bias and Widening Skill Gaps

A study analyzing over five years of data from 52,000 users on an online chess platform reveals that individuals who choose to seek AI feedback are typically more motivated and possess higher initial skills. These users also demonstrate more productive engagement with the technology. This self-selection creates an illusion of AI effectiveness, as apparent learning gains vanish when accounting for endogenous motivation. The research, documented in arXiv preprint 2409.18660v2, investigates how this endogenous choice shapes both individual learning and collective outcomes. The selection mechanism leads to significant population-level consequences, disproportionately benefiting already motivated, higher-skilled individuals. Consequently, widespread AI access may exacerbate existing skill disparities rather than democratize learning opportunities. The findings challenge assumptions about AI's neutral educational benefits, highlighting hidden costs in feedback systems.