Zero-CoT Truncation Exposes LLM Reasoning Contamination
Researchers have proposed a new method, Zero-CoT Probe (ZCP), to detect evasive data contamination in large language models (LLMs). The study, published on arXiv (2605.21856), reveals that a model's generated reasoning steps can mask memorization of benchmark data. ZCP truncates the Chain-of-Thought (CoT) process to expose shortcut mappings, isolating memorization from genuine problem-solving. This addresses the issue of malicious publishers paraphrasing benchmark data to artificially boost leaderboard performance, a problem current detection methods struggle with.
Key facts
- arXiv paper 2605.21856 introduces Zero-CoT Probe (ZCP)
- ZCP detects evasive data contamination in LLMs
- Method truncates Chain-of-Thought to expose memorization
- Evasive contamination involves paraphrasing benchmark data
- Current detection methods fail against stealthy contamination
- Model reasoning steps can mask underlying memorization
- ZCP is a black-box detection method
Entities
Institutions
- arXiv