Chain-of-Thought Prompting: Serial-Depth Bottleneck in LLM Reasoning

A recent empirical investigation questions the belief that chain-of-thought (CoT) prompting consistently enhances reasoning in large language models (LLMs). This research, found on arXiv (2608.09942), presents the idea of a 'serial-depth bottleneck' within transformer architectures. Although the formal H_dp bandwidth limit (Chen et al., 2024) is applicable only asymptotically, the authors suggest it indicates a real constraint: extensive serial computation tasks must be externalized, which is what CoT accomplishes. The study reveals a serial-depth gradient within benchmarks: accuracy without CoT declines steadily with increased serial depth, while CoT remains largely unaffected. Experiments on three instruction-tuned models (Qwen-2.5-7B/32B, Llama-3.1-8B) across five NLP benchmarks show that for high-depth P-complete tasks like GSM8K and MATH, CoT improves performance by +54 to +68 percentage points. Conversely, for shallow TC^0 tasks such as MMLU and ARC, CoT offers no notable advantages. These results indicate that CoT is particularly effective for tasks that surpass a transformer's single-pass serial capacity, informing its practical application and potentially enhancing efficiency.