Concern Alignment Framework for Evaluating AI Peer Reviews

There's a new diagnostic method called concern alignment that looks at AI-generated peer reviews by examining the level of concern rather than just the final judgment. This approach uses a match graph to create a bipartite alignment between official concerns and those produced by AI, categorizing them by type, severity, and how they’re handled after rebuttals. It also sets up an evaluation ladder that goes from basic accuracy to more nuanced assessments like concern detection and decision-aware calibration. A preliminary study of four public AI review systems in six setups reveals that the quality of alignment isn't just about detection. You can find this research on arXiv under the identifier 2604.19998.