LLM Schema Descriptions Can Override Prompts, Study Finds

A recent study published on arXiv (2608.08254) explores the impact of structured-output schema descriptions on the performance of large language models (LLMs) in classification tasks. The investigation involved ten different model configurations from two providers, examining whether the placement of classification-label definitions in the system prompt, user prompt, or schema description yields better results. Findings reveal that schema descriptions do not consistently outperform prompts; specifically, for GPT-4.1 and GPT-5.4 without reasoning, schema placement lagged behind system prompts by 11-13 percentage points. Furthermore, when prompts and schemas conflicted, accuracy suffered significantly, with drops of 5-45 points. Notably, Claude Haiku 4.5 plummeted from 52.5% to 7%, while GPT-5.5 fell from 100% to 73%, indicating that schema instructions can override those in prompts. This study underscores the importance of structured output in AI system design, especially where it serves as the standard approach for data labeling and information extraction.