MMShopBench: New Benchmark for Multimodal Shopping Agents
A new benchmark, MMShopBench, has been developed by researchers to assess multimodal, multi-turn shopping agents. It is based on authentic shopping logs that have undergone thorough cleaning and manual annotation to establish accurate labels for purchase intent and essential product criteria. Unlike prior benchmarks that focus solely on text or synthetic requests, MMShopBench reflects the intricacies of actual shopping experiences, where users communicate their needs through a mix of images and dialogue. Agents must deduce requirements from user images and conversations, retrieve product candidates via both text and image searches, and confirm that each candidate meets all specified criteria by analyzing product images and structured attributes. The evaluation encompasses a variety of open-source and proprietary models, offering a detailed analysis of current AI shopping assistants. This initiative fills a significant void in AI evaluation, as online shoppers increasingly rely on AI assistants capable of interpreting multimodal inputs. The benchmark aims to propel innovations in AI shopping technology, enhancing responsiveness to genuine user demands.
Key facts
- MMShopBench is the first real-log benchmark for multimodal, multi-turn shopping agents.
- It is built from carefully cleaned and manually annotated shopping logs.
- The benchmark provides ground-truth annotations of purchase intent and mandatory product requirements.
- Agents must infer requirements from user images and multi-turn dialogue.
- Agents retrieve candidate products through image and text search.
- Agents verify candidates using product images and structured attributes.
- The evaluation includes representative open-source and proprietary models.
- The benchmark addresses the underrepresentation of complex real-world shopping requirements in existing benchmarks.
Entities
Institutions
- arXiv