NVIDIA Unveils Cosmos-H-Dreams Real-Time Surgical Simulator

NVIDIA has unveiled Cosmos-H-Dreams, a generative simulator for surgical robotics that operates in real-time and is action-conditioned. This new system builds upon the Cosmos-H-Surgical-Simulator, which creates future surgical videos based on initial scenes and robotic actions. Cosmos-H-Dreams refines this capability into a concise student model utilizing the FlashDreams accelerated inference library. It runs on a single NVIDIA RTX PRO 6000 GPU, achieving around 160 frames per second for interactive control by either humans or learned policies. Tailored for the da Vinci Research Kit (dVRK) tabletop suturing, it integrates with CMR Surgical's Versius surgeon controller. The distillation employs a teacher-to-student framework with self-forcing supervision for autoregressive generation. Cosmos-H-Dreams features interfaces like a browser client via WebRTC, a Meta Quest client using WebXR, and direct links to surgical policies, while also allowing for extensibility and custom dataset training. It aims to facilitate offline policy evaluation, synthetic data creation, and potential future uses such as latency-aware telesurgery and interactive surgical rehearsal.