Soft Guidance Outperforms Chain-of-Thought Prompting in Modern LLMs

A recent paper on arXiv (2608.03550) questions the prevailing use of Chain-of-Thought (CoT) prompting as the benchmark for assessing reasoning capabilities in large language models (LLMs). The researchers note that numerous contemporary LLMs naturally produce CoT-style answers when faced with reasoning challenges, leading to a reconsideration of few-shot CoT prompting. Their evaluation of various mid-sized models tackling math problems reveals that models designed for reasoning excel in a straightforward zero-shot context, outperforming few-shot CoT examples and significantly exceeding previously reported outcomes (e.g., Mathstral rises from ~77% to ~84% on GSM8K). Additionally, a zero-shot CoT prompt is adequate for a general-purpose model to exceed the few-shot baseline. The authors attribute this trend to advancements in LLM training, indicating that the need for explicit CoT prompting may diminish as models become more adept at reasoning. These results carry important implications for prompt engineering and evaluation strategies in AI research.