AI Research Exposes Critical Bias in Code Localization Models Through New Diagnostic Benchmark

A study released on arXiv (ID: 2604.16021v1) uncovers a critical weakness in contemporary autonomous software engineering systems. It indicates that current code localization benchmarks are overly reliant on keyword references, such as file paths and function names, leading to what the researchers call the "Keyword Shortcut." This bias prompts AI models to depend on superficial lexical matching instead of authentic structural reasoning regarding code architecture. To tackle this issue, the researchers defined the challenge of Keyword-Agnostic Logical Code Localization (KA-LCL) and introduced KA-LogicQuery, a benchmark that necessitates structural reasoning without naming cues. Testing state-of-the-art methods on this benchmark revealed severe performance declines, highlighting their deficiencies in deterministic reasoning. The team proposed LogicLoc, an innovative framework that merges large language models with robust logical reasoning skills, underscoring the significant shortcomings of current AI systems in software engineering and setting new benchmarks for assessing true reasoning in code analysis.