Inverse Reinforcement Learning for Learning Agents
A new arXiv preprint (2605.09217) formalizes the problem of inferring preferences from a learning agent's behavior, moving beyond standard inverse reinforcement learning (IRL) which assumes optimal human behavior. The authors model the agent as either no-regret or converging to an optimal Boltzmann policy over time. They establish theoretical guarantees for preference learning algorithms in each setting, addressing cases where the human is initially suboptimal. The work aims to improve AI alignment by enabling systems to understand evolving human preferences.
Key facts
- arXiv:2605.09217
- Inverse reinforcement learning (IRL) assumes humans are approximately optimal
- The paper formalizes learning preferences of a learning agent
- A predictor observes a learner acting online
- The learner is modeled as no-regret or converging to optimal Boltzmann policy
- Theoretical guarantees are established for various preference learning algorithms
- The goal is to infer the underlying reward function being optimized
- The human may be learning to act optimally in an environment
Entities
Institutions
- arXiv