Study Reveals Causal Attention Limitations in Language Models' Prompt Order Sensitivity

A research paper published on arXiv demonstrates that large language models show significant performance variations based on prompt structure. In multiple-choice question answering tasks, arranging context before questions and options yields over 14 percentage points higher accuracy than the reverse order. This pattern holds consistently across various models and datasets. The study identifies causal attention mechanisms as the primary cause: when questions and options appear first, the causal mask prevents option tokens from accessing context information, creating an information bottleneck. This architectural limitation makes context invisible to options in certain prompt configurations. The research provides systematic analysis of how attention mechanisms influence model behavior in practical applications. The findings were shared through the arXiv preprint server, which hosts scientific papers in fields including computation and language. The platform's arXivLabs framework enables community collaboration on experimental projects while maintaining values of openness and data privacy.