Risks of automated alignment research for superintelligence

A new paper on arXiv (2605.06390) argues that automating alignment research for artificial superintelligence (ASI) using AI agents could lead to catastrophic safety failures. Even without deliberate sabotage, the plan may produce misleading safety assessments because alignment research involves fuzzy tasks that are hard to supervise and for which human judgment is systematically flawed. Research outputs would contain undetected errors, and correct outputs could be aggregated into overconfident conclusions. The problem is exacerbated by optimization pressure on agent-generated research, making it worse than human-generated alignment work.