SPECTRA: Synthetic Test Collections for Scalable IR Evaluation
SPECTRA serves as a consistent framework for creating synthetic text collections and retrieval test sets, intended to enhance both Cranfield and TREC evaluation methods. It distinguishes between latent topical organization, surface text generation, metadata management, query intent formulation, and deterministic relevance standards. A Python prototype, operating as a single process, produced collections containing as many as 60,000 documents and 9.61 million tokens, featuring adjustable long-tail vocabulary expansion and relevance gradation for 96 queries. This framework fulfills the demand for extensive testing while minimizing dependence on costly human assessments or confidential documents.
Key facts
- SPECTRA generates synthetic corpora up to 60,000 documents and 9.61 million tokens.
- Framework separates latent topical structure, surface text, metadata, query intent, and relevance oracles.
- Designed as a diagnostic complement to Cranfield and TREC evaluation.
- Single-process Python prototype produced graded relevance labels for 96 queries.
- Controllable long-tail vocabulary growth is preserved.
- Aims to stress index construction, ranking latency, query routing, and evaluation tooling.
- Human-judged test collections remain expensive and may be unavailable for private documents.
- SPECTRA is not a replacement for human assessment.
Entities
—