ERUnderstand: Benchmarking VLMs on Entity-Relationship Diagrams
Researchers have introduced ERUnderstand, the first comprehensive benchmark for understanding Entity-Relationship Diagrams (ERDs). It features a collection of 2,960 diagrams drawn from educational resources, real-world schemas, and synthetic examples across various fields and notations, including Extended Entity-Relationship elements. Each diagram comes with a machine-readable format for thorough evaluation. When tested with advanced Vision-Language Models (VLMs), results showed strong performance on basic elements (F1 > 0.74), but a notable drop in accuracy for weak entities (F1 at 0.28), multivalued attributes (0.14 F1), and N-ary relationships (0.07 F1). This study highlights the limitations of current VLMs in understanding structured diagrams.
Key facts
- ERUnderstand is the first large-scale benchmark for structured ER diagram understanding.
- The dataset includes 2,960 diagrams from educational, real-world, and synthetic sources.
- Diagrams cover diverse domains, notations, and Extended Entity-Relationship constructs.
- Each diagram has a standardized machine-readable representation for evaluation.
- State-of-the-art VLMs achieve F1 > 0.74 on common ERD elements.
- Performance drops to 0.28 F1 on weak entities.
- Multivalued attributes achieve only 0.14 F1.
- N-ary relationships score as low as 0.07 F1.
Entities
—