HeraSys Optimizes Concurrent LLM Workflows via Fine-Grained Orchestration
A recent publication on arXiv (2607.22578) presents HeraSys, a system for large language model (LLM) serving that aims to enhance the overall performance of concurrent, multi-tenant workflows. In contrast to current solutions that mainly focus on optimizing within a single workflow, HeraSys emphasizes optimization across workflows by reducing computational redundancy through the merging and reuse of structural nodes. It incorporates a load-aware joint scheduling policy, which dynamically adjusts the execution sequence based on inter- and intra-query priorities. Furthermore, HeraSys employs a resource skewing mechanism alongside adaptive batching and pipeline decomposition to reduce tail latency while ensuring low average latency. This system marks a shift from handling isolated requests to managing high-concurrency workflows in LLM serving.
Key facts
- HeraSys is an LLM serving system for concurrent workflows.
- It eliminates cross-workflow computational redundancy via structural node merging and reuse.
- It introduces a load-aware joint scheduling policy.
- It uses resource skewing, adaptive batching, and pipeline decomposition.
- It mitigates tail latency while maintaining low average latency.
- The paper is on arXiv with ID 2607.22578.
- It addresses inter-workflow optimization, not just intra-workflow.
- The system targets high-concurrency, multi-tenant agentic workflows.
Entities
Institutions
- arXiv