Policy-Invariant Reward Shaping from LLM Feedback: A Framework for Hybrid RL Agents

The arXiv preprint 2608.18008 introduces a novel framework for hybrid reinforcement learning agents that merges a large language model (LLM) planner with a reinforcement learning (RL) controller, framed as a Goal-Augmented Markov Decision Process (MDP). The LLM produces a progress score for each state, acting as a bounded potential function, which maintains the integrity of the optimal policy set despite potential inaccuracies in the LLM's scores. This reliability is crucial for practical applications. The study includes numerical validation on a small MDP, exploring four configurations of the potential function and demonstrating robustness even in extreme scaling scenarios. This preprint, accessible on arXiv, enhances research on the integration of LLMs with RL, pertinent to fields like robotics, dialogue systems, and autonomous decision-making.