Hierarchical BM25: Efficient Billion-Document Lexical Search
A recent study published on arXiv (2608.00229) presents Hierarchical BM25, a technique designed for approximate lexical searches across billions of documents. The authors highlight that a standard BM25 index for over one billion documents requires around 400 GB of storage, necessitating DRAM proportional to the corpus size, and that disk-based retrieval can take between 4 to 12 seconds per query, rendering precise top-k retrieval impractical for interactive use. Hierarchical BM25 trades off exact ranking to maintain fixed limits on memory and latency. It employs a coarse index to determine which of roughly 1,000 topical document groups to query, based on two indicators: the total frequency of query terms in a group and the co-occurrence of sparse informative terms. Exhaustive searches are then conducted within the chosen groups, scoring against about 100 KB of global statistics, ensuring that returned scores match those of the flat index, with approximation limited to the selection process. The memory footprint is approximately 4.4 GB, a substantial decrease from the 400 GB flat index. This method facilitates interactive lexical searches over extensive corpora, which could enhance applications in information retrieval, digital archives, and large-scale data analysis. The paper can be accessed via the provided arXiv link.
Key facts
- Flat BM25 index over one billion documents occupies about 400 GB.
- Serving from disk takes 4-12 seconds per query.
- Hierarchical BM25 gives up exact ranking for fixed bounds on memory and latency.
- Resident coarse index selects from ~1K topical, size-balanced document groups.
- Selection uses two signals: total frequency of query terms within a group, and co-occurrence of informative terms in a document.
- Selected groups are searched exhaustively and scored against ~100 KB of global statistics.
- Returned scores equal the flat index's score; approximation is confined to selection.
- Resident footprint is ~4.4 GB.
Entities
—