Text-Guided Dual-Gaze Prediction for Object-Level Driver Attention

Researchers have rolled out a new framework designed to improve how autonomous vehicles predict where to focus their attention on objects. They created the G-W3DA dataset by blending a large multimodal language model with the Segment Anything Model 3 (SAM3). This combination allows them to break down broader heatmaps into specific object-level masks using rigorous cross-validation. This development addresses the limitations of existing datasets, which typically only provide global scene-level gaze. Such limitations can lead to issues like text-vision decoupling and visual biases in Vision-Language Models (VLMs). The study details a thorough method that covers everything from creating the data to designing the model.