VibeWorlding: New Benchmark for Multimodal Agents in 3D World Construction
Researchers have introduced an innovative framework named VibeWorlding, aimed at evaluating and training multimodal agents that can generate interactive 3D open worlds based on user feedback. This new approach addresses the lack of systematic evaluation methods, as earlier assessments were too basic. VibeWorlding empowers agents to interpret user intent, organize scenes, apply 3D tools, and handle multi-turn conversations with various types of feedback. To support this, the team developed VWE-BENCH, which includes 2,616 high-quality 3D assets, 323 human-annotated seed worlds, and 6,828 multimodal user queries, split into verified and unverified categories. They also created VibeWorlding-Gym as a training space for these agents. The findings are published on arXiv (arXiv:2608.15265v1) and could significantly improve AI's ability to understand and create complex 3D environments, impacting gaming, virtual reality, and digital art.
Key facts
- VibeWorlding is a unified framework for benchmarking and training multimodal agents for 3D open world construction.
- The framework allows agents to infer user intent, plan layouts, invoke 3D tools, and reflect on multimodal feedback.
- VWE-BENCH includes 2,616 high-quality 3D assets, 323 human-annotated seed 3D worlds, and 6,828 reverse-synthesized multimodal user queries.
- Queries are split into verified (with ground-truth) and unverified (with rubrics) categories.
- VibeWorlding-Gym is a training environment for agents.
- The paper is available on arXiv with identifier arXiv:2608.15265v1.
- The research addresses the lack of systematic evaluation for multimodal agents in 3D world construction.
- Potential applications include gaming, virtual reality, and digital art.
Entities
Institutions
- arXiv