LLM Screening in Systematic Reviews: Batch Effects Outweigh Prevalence Metadata

A new study from arXiv examines the performance of large language models (LLMs) in imbalanced binary classification tasks, specifically focusing on study screening for systematic reviews. The research, conducted across five reviews, compared individual and batch processing methods, with and without the inclusion of prevalence metadata. The findings indicate that prevalence metadata has limited influence and does not improve performance. In contrast, batch processing led to larger behavioral changes that varied depending on class prevalence. Notably, aggregate and item-level analyses did not always align, suggesting that batch processing should be evaluated not only for cost efficiency but also for its impact on decision-making behavior. The study contributes to the growing body of research on LLM applications in academic and medical contexts, highlighting the need for careful consideration of processing methods in systematic review workflows.