Chinese AI Startup Spirit Accused of Benchmark Hacking After Briefly Topping Robotics Ranking
Spirit AI, a Chinese startup focused on physical AI, briefly held the leading position on the RoboArena robotics benchmark, overtaking Nvidia, but was subsequently disqualified due to accusations of 'benchmark hacking'. This incident underscores the fierce competition between the US and China in AI advancements and the difficulties in assessing autonomous technologies. The benchmark's developers revised their evaluation process, excluding Spirit's model and others, including a fourth-place model from the Chinese company X Square Robot. Pranav Atreya, a PhD student at UC Berkeley and a key author of the project, revealed on X in June that evaluations from organizations involved in benchmark manipulation were 'retroactively removed', though no specific companies were identified. Spirit had referred to the ranking as the 'Olympics' of embodied intelligence in North America.
Key facts
- Spirit AI's Spirit v1.6 briefly topped the RoboArena robotics benchmark, surpassing Nvidia.
- The model was removed after the benchmark's creators overhauled methodology due to 'benchmark hacking'.
- X Square Robot, a Chinese startup, also had its model removed from the rankings.
- Pranav Atreya, a lead author and UC Berkeley PhD student, announced the removals on X in June.
- The controversy underscores US-China competition in AI development.
- The benchmark was dubbed the 'Olympics' of embodied intelligence by Spirit AI.
- Atreya did not name specific companies in his statement.
Entities
Institutions
- Spirit AI
- Nvidia
- RoboArena
- X Square Robot
- UC Berkeley
Locations
- China
- United States