VitaBench 2.0 Evaluates Personalized AI Agents in Long-Term Interactions
VitaBench 2.0 has been launched by researchers as a new benchmark aimed at assessing the personalized and proactive behaviors of large language model (LLM) agents during extended user interactions. In contrast to current benchmarks that prioritize reasoning and tool utilization, VitaBench 2.0 tackles the issue of deducing user preferences from sporadic daily exchanges. The tasks are structured as sequences arranged in chronological order for each user, compelling agents to consistently identify and apply preferences found within diverse interactions. The goal of this benchmark is to enhance the evolution of LLM agents, enabling them to engage more effectively by grasping user needs beyond what is explicitly stated.
Key facts
- VitaBench 2.0 is a new benchmark for evaluating personalized and proactive agent behavior.
- It focuses on long-term user interactions with temporally ordered tasks.
- The benchmark addresses the gap in existing agent benchmarks that overlook user preference inference.
- Tasks require agents to extract preferences from fragmented and heterogeneous interactions.
- The work is published on arXiv with identifier 2605.27141.
Entities
Institutions
- arXiv