Higher Observation Fidelity Hurts Embodied LLM Problem Solving

A new study on arXiv (2605.20072) reveals that large language models (LLMs) don’t perform as well in robotic systems when they have perfect data compared to using raw RGB images. The researchers tested LLM agents with the Lockbox, which is a complex puzzle with hidden connections, and looked at RGB, RGB-D, and perfect symbolic data in a real-world robotic setting. Interestingly, the agents using just raw RGB images outperformed those with perfect data. Furthermore, simulations showed that tweaking perceived outcomes randomly actually boosted performance, hitting an optimal flip probability of 40%, which led to a 2.85-fold increase in success. This raises doubts about the idea that better observation quality always helps in these tasks.