LLM Code Review: Policy Shifts Alter Risk Probabilities by Up to 16.9 Points

A new study from arXiv (2608.02677) examines how LLM code reviewers handle the separation between probability estimation and decision-making. The researchers tested four deployed reviewer interfaces using 15,792 responses on 720 candidate patches, each paired with a passing and failing test harness from 360 repository issues. They found that when the cost of false accepts was increased from equal to 10:1, reported failure probabilities shifted by 13.6 to 16.9 percentage points on average, even though the patch and monitor evidence were fixed. This indicates that the prompts used to elicit probabilities are not neutral; they are influenced by the decision policy. Moreover, the actions returned under the high-cost prompt were worse than simply rejecting all patches for every reviewer. However, applying the high-cost rule to probabilities elicited under equal costs reduced loss for all four systems, suggesting that the probability elicitation process itself contributes to excess loss. The study also evaluates a modular pipeline that separates probability elicitation from decision-making, potentially offering a more robust approach. The findings highlight the importance of separating risk assessment from action selection in LLM-based code review systems, with implications for software engineering practices and AI safety.