CuraWeb: Joint Optimization of Quality, Redundancy, and Diversity for LLM Pretraining Data

CuraWeb introduces a novel approach to web-scale pretraining data curation, moving away from traditional linear pruning towards a method that jointly optimizes quality, redundancy, and diversity. This framework integrates dual-track cleaning—both rule-based and model-driven—with hybrid deduplication techniques, utilizing n-gram and semantic methods. A multi-objective sampler ensures a balance between informational quality and distributional variety. When applied to Common Crawl, CuraWeb generates a 2T-token English corpus. This methodology overcomes the shortcomings of selective filters such as FineWeb-Edu and DCLM, which limit distributional diversity and overlook long-tail knowledge, thereby maximizing the potential of the open web. The findings are detailed in arXiv:2607.22662v1.