Sampled-BPE: Auditing Chinese Web Corpora for LLM Pollution
A recent study published on arXiv introduces a novel auditing technique called Sampled-BPE. This innovative approach is aimed at identifying contamination within large Chinese datasets commonly utilized in training large language models (LLMs). By sampling a small fraction of the data and employing a BPE tokenizer, researchers effectively detected polluted tokens, managing the complexities of scale and adapting to the dynamic nature of web pollution. The method resulted in notable enhancements, achieving a speed increase of 148.4 times and reducing memory demands by 35.8 times, with a mere 4.25% error rate across assessments of multiple Chinese corpora from 2021 to 2026.
Key facts
- Paper arXiv:2608.10678 introduces Sampled-BPE, a token-level auditing pipeline for Chinese web corpora.
- Sampled-BPE samples a small subset and trains a BPE tokenizer to surface polluted tokens.
- Experiments show 148.4x speedup and 35.8x memory reduction with only 4.25% relative error.
- The pipeline was applied to 11 open Chinese corpora and 6 Chinese Common Crawl snapshots from 2021 to 2026.
- The audit reveals widespread but uneven pollution across open corpora.
- Chinese web content is highly polluted and temporally shifting.
- The paper addresses challenges of scale, granularity, and implicit pollution in Chinese web corpora.
- The research is relevant for LLM training data quality and auditing.
Entities
Institutions
- arXiv
- Common Crawl
Locations
- China