RealICU Benchmark Tests LLM Reasoning on Long ICU Data

A new benchmark called RealICU has been developed by researchers to assess large language models (LLMs) using extensive ICU data, where labels are assigned following a thorough review of complete patient histories by senior physicians. This benchmark overcomes the shortcomings of current ICU benchmarks that rely on historical clinician actions as definitive, which can lead to suboptimal results due to missing information. RealICU outlines four tasks driven by physician needs: evaluating Patient Status, identifying Acute Problems, suggesting Recommended Actions, and recognizing Red Flag actions that could lead to unsafe outcomes. Each patient trajectory is segmented for assessment. This research is available on arXiv (2605.13542).