GeoRepEval Framework Assesses LLM Robustness Across Geometry Problem Representations

Researchers have created a new evaluation framework named GeoRepEval to assess the performance of large language models (LLMs) on geometry problems presented in various mathematical formats. They observed that current benchmarks often evaluate LLMs with fixed problem structures, which overlooks failures arising from different representations. GeoRepEval tests eleven language models against 158 specially selected geometry problems, examining them through parallel formats such as Euclidean, coordinate, and vector representations. The framework focuses on three main criteria: correctness, invariance, and consistency at the problem level, utilizing various statistical techniques. Among these is the Invariance@3 metric, which breaks down accuracy into robust and fragile elements. The findings reveal significant variability in LLM performance based on problem expression. This research, documented in paper arXiv:2604.16421v1, fills a critical gap in understanding model robustness in mathematical contexts.