Hierarchical Adaptive Refinement Accelerates Policy Synthesis in Large MDPs

A novel strategy for synthesizing policies in Markov decision processes (MDPs) employs hierarchical adaptive refinement to manage extensive state spaces. This technique iteratively identifies the most sensitive areas, optimizing both accuracy and efficiency. The resulting policy is demonstrated to be nearly optimal based on conventional assumptions, with errors limited by local solver tolerances and boundary discrepancies. In practical applications involving MDPs with as many as 1 million states, this method provides up to a 2× increase in speed compared to PRISM, presenting a viable solution for real-world policy synthesis in software-heavy systems such as software product lines and robotics.