CoT-Core: Training-Free Coreset Selection Accelerates LLM Evaluation
A new method for accelerating the evaluation of large language models (LLMs) has been introduced in a preprint on arXiv. The method, called CoT-Core, addresses the high computational cost of evaluating LLMs during development. Traditional coreset selection methods either require extensive historical data (a 'cold start' problem) or rely on surface lexical similarity, missing the underlying reasoning structure of tasks. CoT-Core is a training-free approach that uses zero-shot Chain-of-Thought (CoT) reasoning trajectories to cluster questions by logical equivalence rather than text similarity. This is achieved by prompting LLMs to generate reasoning paths and projecting them into a latent space. Experiments on benchmarks including GSM8K, MMLU, MMLU-Pro, and GPQA show that CoT-Core significantly reduces evaluation costs while maintaining high-fidelity scores. The paper is available at arXiv:2608.00014.
Key facts
- CoT-Core is a novel training-free coreset selection framework for LLM evaluation.
- It uses zero-shot Chain-of-Thought reasoning trajectories to cluster questions by logical equivalence.
- The method addresses the 'cold start' bottleneck of existing methods like Item Response Theory.
- Experiments were conducted on GSM8K, MMLU, MMLU-Pro, and GPQA benchmarks.
- CoT-Core drastically reduces evaluation costs while maintaining high-fidelity scoring.
- The paper is published on arXiv with identifier 2608.00014.
- The approach is training-free, meaning it does not require additional model training.
- It projects reasoning paths into a latent space to cluster questions.
Entities
Institutions
- arXiv