AMIE (Video) Achieves Expert-Level AI in Real-Time Clinical Video Consultations
A recent investigation published on arXiv (2608.09861) introduces AMIE (Video), a multi-agent system based on Gemini that achieves expert-level proficiency in real-time clinical video consultations. This marks the inaugural instance of AI demonstrating clinician-level skill in audio-visual medical interactions. The system incorporates low-latency dialogue, clinical reasoning, and real-time audio-visual perception, overcoming the shortcomings of text-based AI that overlook critical perceptual elements. The researchers created a taxonomy and automated assessments for clinical audio-visual signals in telehealth, which informed the system's development. In a randomized Objective Structured Clinical Examination (OSCE), AMIE (Video) exhibited performance on par with human clinicians, showcasing AI's potential to improve telehealth through natural communication and effective evaluation of non-verbal cues, especially for patients unable to express symptoms in writing.
Key facts
- AMIE (Video) is a Gemini-based multi-agent system for real-time clinical video consultations.
- It is the first demonstration of expert-level AI in real-time clinical video consultations.
- The system integrates low-latency dialogue, clinical reasoning, and real-time audio-visual perception.
- A taxonomy and automated evaluations for clinical audio-visual cues in telehealth settings were established.
- In a randomized Objective Structured Clinical Examination (OSCE), AMIE (Video) matched clinician-level performance.
- The study addresses limitations of text-based AI in medical consultations.
- The research is available on arXiv under the identifier 2608.09861.
- The study emphasizes the importance of audio-visual interaction in patient-physician consultations.
Entities
Institutions
- arXiv