TRACE-Router: Task-Level Routing Framework for Agentic AI
TRACE-Router, a novel routing framework, tackles the discrepancy between per-call LLM routing and the outcomes at the task level in agentic applications. While current routers make separate decisions for each LLM call, agentic workflows rely on delayed metrics for task-level quality. TRACE-Router allocates a model to each task upon admission via a contextual bandit, links all following LLM calls to that specific backend, and refines its policy based on the terminal reward of the task, considering both accuracy and latency. This method utilizes delayed feedback to develop adaptive routing strategies. The research paper can be found on arXiv.
Key facts
- TRACE-Router is a task-level routing framework for large language models.
- It addresses the mismatch between per-call routing and delayed task-level outcomes in agentic AI.
- It assigns each task to a model once at admission using a contextual bandit.
- All subsequent LLM calls for a task are pinned to the selected backend.
- Policy updates use the task's terminal reward, jointly accounting for accuracy and latency.
- The framework leverages delayed task feedback to learn adaptive routing policies.
- The paper is published on arXiv with ID 2607.22465.
- TRACE-Router stands for Task-Consistent and Adaptive Online Routing.
Entities
Institutions
- arXiv