MedLoCoMo: Benchmark for Long-Context Medical Dialogue with LLMs
MedLoCoMo is a new benchmark designed to evaluate large language models on patient-specific clinical reasoning across multiple medical admissions. Built from deidentified MIMIC-IV and MIMIC-IV-Note records, it constructs admission-level clinical packets, synthesizes grounded doctor-patient conversations, and generates evidence-linked QA items across single-admission, cross-admission, and adversarial unanswerable settings. The benchmark includes 100 patient timelines with an average of 1,669.8 turns, 29.7 sessions, and 74,512.2 tokens per conversation. Initial evaluations show that cross-admission reasoning is consistently more difficult than localized evidence use, even for models with long context windows or external memory and retrieval methods. The code and data are available on arXiv.
Key facts
- MedLoCoMo is a Medical Long-Context Memory benchmark for multi-admission clinical reasoning.
- Built from deidentified MIMIC-IV and MIMIC-IV-Note records.
- Contains 100 patient timelines averaging 1,669.8 turns, 29.7 sessions, and 74,512.2 tokens per conversation.
- Includes single-admission, cross-admission, and adversarial unanswerable QA settings.
- Cross-admission reasoning is harder than localized evidence use for evaluated baselines.
- Even long-context models and retrieval-augmented methods struggle with cross-admission tasks.
- Code and data are publicly available on arXiv.
- Addresses a gap in existing medical QA benchmarks which focus on short context or single documents.
Entities
Institutions
- MIMIC-IV
- MIMIC-IV-Note
- arXiv