NAVSIM v2.2 Scoring Audit Reveals Shared Rollout Failures

A recent study published on arXiv (2608.04896) evaluates the original scene single-stage scoring system of NAVSIM v2.2, indicating potential issues with defensive driving scores when reference-conditioned forgiveness is utilized. The evaluation revealed that, under the documented-stack condition affecting the numerical backend, both the route-blind Ignore-All probe and the route-aware actor-blind probe outperform human replay and PDM-Closed across the entire 12,146-token navtest split. Additionally, a new installation based on the public specification exhibits rollout divergence on a fixed 32-token diagnostic set. The authors contend that defensive driving scores hold value only when they differentiate between policies that acknowledge surrounding actors and those that do not. This problem arises when agent and reference share an unstable rollout transformation, leading to widespread compliance credit issues. The audit includes same-source dependency stack control and an exact-input diagnostic. The paper was released by researchers on August 26, 2025.