Blind Expert Verification Challenges Human Consensus as Ground Truth for LLM Coding

A recent study published on arXiv (2607.28890) investigates the premise that human coding serves as the benchmark for assessing LLM-assisted qualitative coding. In this research, five LLM systems and three trained human coders utilized a 72-item hierarchical codebook to analyze 2,560 messages from educators on a K-12 AI platform. An independent domain expert conducted blind evaluations of 855 pairwise code comparisons, treating both human and machine outputs equally. The findings revealed a discrepancy between the two evaluation methods: the mean Jaccard index for human-LLM agreement was 0.30, significantly lower than the human-human agreement of 0.52. However, the blind evaluator found human and LLM coding to be similarly acceptable (51.5% vs. 48.5%, p = 0.537). This indicates that traditional agreement metrics might not accurately reflect true quality, challenging the notion that human consensus represents definitive truth. The implications of this study are particularly pertinent as AI becomes increasingly integrated into qualitative research and content analysis in educational settings.