MBA-Bench: First Multimodal Benchmark for Business Ideation Agents
A new benchmark called MBA-Bench has been launched by researchers, marking the first multimodal tool aimed at training and assessing business ideation agents that utilize large language models (LLMs). Unlike current text-only methods, this benchmark integrates visual elements crucial for practical business scenarios. MBA-Bench features 30,000 samples across six different domains, each presenting unique visual data that text alone cannot fully express. The creation of this benchmark involved automatically generating captions for images and employing GPT-4o to produce five reference ideas for three business questions, aided by retrieval query generation and market evidence synthesis. Agents are assessed based on six business-related criteria using MLLM-as-a-Judge, with two configurations: MBA-b (blind) and MBA-k (known), depending on the visibility of criteria. This benchmark seeks to enhance agentic systems in business ideation through a standardized evaluation framework.
Key facts
- MBA-Bench is the first multimodal benchmark for business ideation agents.
- It contains 30,000 samples across six domains.
- Each domain has distinct visual cues not fully conveyed by text.
- Images are automatically captioned and GPT-4o generates reference ideas.
- Five reference ideas are generated for each of three business questions.
- Evaluation uses MLLM-as-a-Judge across six business-oriented criteria.
- Two settings: MBA-b (blind) and MBA-k (known) for hidden or disclosed criteria.
- The benchmark is introduced in arXiv paper 2608.11616.
Entities
Institutions
- arXiv