AI Benchmarks as Instruments of Structural Injustice

A recent study published on arXiv (2608.15326) posits that AI benchmarks function not as impartial assessment tools but as socio-technical constructs influencing competition, authority, and research focus. The research, conducted by unnamed scholars, utilizes Iris Marion Young's concepts of oppression and structural injustice to critique current benchmarking methods. It suggests that these practices may reinforce systemic harms, reflecting four of Young's identified 'faces of oppression.' The authors claim that benchmarking culture contributes to structural injustice, as detrimental effects arise from normalized practices and network dynamics. The paper emphasizes that benchmarks create standardized evaluations, leading to leaderboards that bestow prestige, citations, trust, and institutional power on top performers. As expenses for AI development increase, these accolades become concentrated among influential, industry-backed labs, worsening disparities. The paper, categorized as 'new,' does not offer solutions but frames benchmarking as a structural challenge, contributing to ongoing discussions about equity and power in AI, particularly for artists and cultural organizations working with AI technologies.