AI Co-Clinician: Real-Time Medical AI with Audio-Visual Reasoning
Researchers introduce AI co-clinician, a conversational AI system that processes live audio-visual data from patient consultations to inform real-time clinical decisions. Built on Gemini's low-latency voice and video capabilities, it uses a dual-agent architecture balancing deep clinical reasoning with natural dialogue speed. A video-based telemedicine interface was tested with 20 standardized outpatient scenarios requiring auditory and visual reasoning. Evaluation used the novel TelePACES criteria and case-specific rubrics in a randomized, interface-blinded crossover simulation study with 120 encounters involving 10 internal medicine residents.
Key facts
- AI co-clinician is a conversational AI system for real-time clinical decisions.
- It processes continuous streams of audio-visual data from live patient conversations.
- Built on Gemini's low-latency voice and video processing capabilities.
- Dual-agent architecture balances deep clinical reasoning with low latency.
- Tested via a video-based telemedicine interface.
- 20 standardized outpatient scenarios were crafted.
- TelePACES evaluation criteria and case-specific rubrics were designed.
- Randomized, interface-blinded crossover simulation study with 120 encounters and 10 internists.
Entities
Institutions
- arXiv