MecEng: New Benchmark Tests LLMs on Mechanical Engineering and Spatial Geometry
A new automated benchmark called MecEng has been introduced to evaluate how well Large Language Models (LLMs) understand mechanics and spatial geometry, which the creators define as 'mechanical engineering awareness.' While LLMs are proficient in traditional coding and math tasks, their ability to create multibody simulation models from text descriptions hadn’t been assessed before. MecEng addresses this gap with an automated system that includes 84 tasks divided into three levels of complexity. These tasks involve rigid-body systems with joints and flexible multibody systems that require accurate 3D geometry generation and other complex processes. The benchmark aims to thoroughly test LLMs’ understanding of mechanical engineering principles. You can check out the research paper on arXiv with the ID 2608.14615.
Key facts
- MecEng is a fully automated benchmark for evaluating LLMs on mechanical engineering awareness.
- The benchmark includes 84 generic tasks across three difficulty levels.
- Tasks range from rigid-body systems to flexible multibody systems requiring 3D geometry generation and finite-element meshing.
- The pipeline uses Netgen for geometry generation and Exudyn for building multibody system models.
- Verification against expert ground truth includes system-graph isomorphism.
- The paper is available on arXiv with identifier 2608.14615.
- The benchmark addresses a gap in evaluating LLMs beyond code and math benchmarks.
Entities
Institutions
- arXiv