New Framework Measures LLM Reliability for Social Science Annotation Tasks

A new methodological framework called Inter-Prompt Reliability (IPR) has been introduced to assess the stability of large language model outputs when used for annotation in computational social science. The framework evaluates how consistently LLMs perform across semantically equivalent but linguistically varied prompts, drawing inspiration from traditional Inter-Rater Reliability measures. Researchers measured IPR using Pairwise Agreement Rate and its distribution to capture both consistency and stochastic behavior in model responses. The evaluation tested this framework on two distinct annotation tasks: the interpretative TREC task and the knowledge-anchored Politifact task. Results revealed substantial stochastic variation in LLM performance on interpretative tasks, while models appeared more stable when handling knowledge-based annotation work. The study further demonstrated that employing majority voting across multiple prompts significantly enhances reproducibility and reduces variance in LLM outputs. These findings suggest methodological considerations for researchers using LLMs in social science labeling applications.