LLMs Outperform Fine-Tuned Models on Rare Suicide Circumstance Extraction

A recent study published on arXiv presents a 'Complexity Score' algorithm designed to assess when elaborate prompts enhance the performance of large language models (LLMs) compared to simple name-only prompts for extracting structured data from death investigation narratives. This research examines 25 complex inferential scenarios sourced from the National Violent Death Reporting System (NVDRS), evaluating LLMs against a fine-tuned version of RoBERTa. The findings indicate that LLMs significantly excel in low-prevalence situations with limited training data. The proposed framework is applicable across cutting-edge LLMs, including GPT-5.2, Gemini 2.5 Pro, and Llama-3 70B. Notably, suicide continues to be a primary cause of death in the United States, with many scenarios necessitating semantic inference beyond mere keyword matching.