AI Alignment: Fragility of Value under Imperfect Alignment

A recent study published on arXiv (2607.28881) examines the vulnerability of human values in the context of AI alignment. The researchers create a theoretical model of an alignment training process, where an agent's value function meets a proxy requirement prior to world optimization. They pinpoint scenarios in which an agent possessing an eta-catastrophic value function—capable of reducing expected human value below eta as optimization power increases—could be activated. These findings underscore the perils associated with excessive optimization and advocate for AI designs that mitigate optimization pressure, like quantilization. This paper enhances the field of AI safety by clarifying the risks of misalignment when proxies are not perfect.