New AI Research Paper Analyzes LLM Failures Using Contrastive Attribution Methods

A recent study presents contrastive attribution as an effective method for examining failures in Large Language Models within realistic benchmarks. The paper, available on arXiv with the identifier 2604.17761v1, fills a void in interpretability research that has mainly concentrated on simplified settings or brief prompts. The authors define failure analysis via contrastive attribution, linking logit differences from incorrect to correct output tokens to particular input tokens and internal model states. They also created a streamlined extension that allows for the generation of cross-layer attribution graphs for long-context inputs. Through this framework, the researchers performed systematic empirical evaluations across various benchmarks, analyzing attribution patterns across different datasets, model sizes, and training checkpoints. The findings indicate that token-level contrastive attribution can provide valuable insights into model behavior in real-world applications, marking a significant advancement in understanding LLM failures outside of artificial test conditions.