StreamReason-Bench: LLMs Struggle with Event-Time Stream Processing

StreamReason-Bench is a newly established benchmark designed to assess large language models (LLMs) in their grasp of event-time stream-processing semantics. Detailed in an arXiv paper (2608.12348), this benchmark challenges models to emulate an event-time stream processor. Given a windowed query alongside a stream of out-of-order events, the model is tasked with identifying which windows activate (along with their aggregates) and which events are considered too late. The scoring key is based on a compact reference implementation of Dataflow-model semantics, enabling precise grading with partial-credit row-F1 without the need for an engine. Featuring 600 generated items that encompass tumbling, hopping, session, and processing-time windows, findings indicate that models struggle with event-time reasoning. When prompted directly, none of the models that adhere to the instructions achieve an exact match above 34%. However, employing chain-of-thought (CoT) prompting significantly enhances performance for several models, with GPT-4o demonstrating marked improvement. The paper underscores a significant shortcoming in LLMs for stream-processing tasks, which are increasingly assigned to them in streaming systems for tasks such as writing pipelines, triaging alerts, and analyzing logs. This benchmark serves as a comprehensive evaluation tool for this developing application area.