Study Reveals LLM Vulnerabilities to Chain-of-Thought Perturbations

A comprehensive empirical evaluation examines the robustness of Large Language Models (LLMs) when their reasoning processes are disrupted. Researchers tested 13 models across a wide parameter range against five specific types of Chain-of-Thought (CoT) perturbations: MathError, UnitConversion, Sycophancy, SkippedSteps, and ExtraSteps. The study, documented in arXiv:2603.03332v3, focuses on mathematical reasoning tasks where corruptions are intentionally injected into intermediate reasoning steps. Findings show heterogeneous vulnerability patterns, with MathError perturbations causing the most severe accuracy degradation in smaller models, sometimes reaching 50-60% loss. UnitConversion errors proved persistently challenging, maintaining over a 5% accuracy loss even for midsized models. The research highlights that while CoT prompting is foundational for eliciting reasoning from LLMs, its resilience to such structured corruptions remains poorly understood. The evaluation spans models across three orders of magnitude in parameter count, revealing that scaling benefits strongly mitigate some vulnerabilities but not others. This work provides a structured taxonomy for assessing reasoning robustness in AI systems.