Collaborative Distributed Inference System for Efficient QoS-Aware Autoscaling

A recent research article introduces a collaborative distributed inference framework that integrates dedicated infrastructure with resources provided by users to meet the increasing demand for AI inference. This framework employs dedicated resources to ensure a baseline capacity for quality of service (QoS), while user-contributed resources manage demand surges without necessitating a proportional increase in centralized infrastructure. The authors create a high-dimensional generative Markov model featuring structured temporal factorization to capture the stochastic and dynamic relationships among users, resources, tasks, and policies. This model facilitates simulation and lays the groundwork for optimizing task scheduling and QoS-aware resource allocation. Evaluations demonstrate that distributed scheduling policies can effectively leverage volunteered resources, minimizing the need for centralized infrastructure expansion while preserving QoS. The study can be accessed on arXiv under identifier 2608.11840.