Kalypso: Relational LLM Serving for Semantic Query Processing

A recent preprint on arXiv (2607.23815) presents Kalypso, a system for relational LLM serving aimed at enhancing semantic query processing. Large language models are increasingly utilized as semantic operators for various tasks, including filtering, extracting, ranking, joining, and transforming unstructured data. Current systems for semantic query processing often depend on request-centric LLM serving, which does not consider the query plan, resulting in missed performance improvements. Kalypso introduces a concept known as relational LLM serving, which ensures LLM serving is aware of the semantic structure of queries while maintaining query semantics and output precision. Its primary innovation involves pipelined execution among semantic operators, allowing for the reuse of KV-cache states from intermediate tuples rather than recalculating them. Kalypso also offers an API for semantic query plans and employs an adaptive, memory-aware scheduling algorithm to tackle a novel online scheduling challenge that optimizes pipelined operator execution. The paper has been revised (replace-cross).