BM25 Ranking Algorithms

0.2.2 · active · verified Thu Apr 09

Provides various BM25 algorithms (BM25Okapi, BM25L, BM25Plus) for document ranking based on a corpus of tokenized documents. It's currently at version 0.2.2 and appears to have an infrequent release cadence, with the latest update adding support for non-iterable corpuses.

Warnings

gotcha The BM25 algorithms expect a pre-tokenized corpus (a list of lists of strings) and tokenized queries, not raw strings. Each sub-list represents a document's tokens.
Fix: Ensure your corpus (list of documents) and queries are tokenized into lists of words (e.g., using `doc.split(' ')` or an NLP tokenizer) before passing them to the BM25 constructor or `get_scores`/`get_top_n` methods.
gotcha The PyPI package name for installation is `rank-bm25` (with a hyphen), but the Python module you import into your code is `rank_bm25` (with an underscore).
Fix: Use `pip install rank-bm25` for installation and `from rank_bm25 import ...` for importing classes like `BM25Okapi`.
gotcha Prior to version 0.2.2, passing non-iterable corpuses (e.g., generators) to the BM25 constructor was not officially supported and could lead to unexpected behavior or errors. While 0.2.2 added support, be mindful of generator behavior.
Fix: For versions prior to 0.2.2, ensure your corpus is a concrete list of tokenized documents. For 0.2.2+, generators are supported, but remember they are single-pass and may need to be re-initialized if reused.

Install

pip install rank-bm25 Install stable version

Imports

BM25Okapi
```
from rank_bm25 import BM25Okapi
```
The PyPI package is 'rank-bm25' but the module name for import is 'rank_bm25' (with an underscore).
BM25L
```
from rank_bm25 import BM25L
```
The PyPI package is 'rank-bm25' but the module name for import is 'rank_bm25' (with an underscore).
BM25Plus
```
from rank_bm25 import BM25Plus
```
The PyPI package is 'rank-bm25' but the module name for import is 'rank_bm25' (with an underscore).

Quickstart

This example demonstrates how to initialize BM25Okapi with a tokenized corpus and then retrieve scores and top-N documents for a given tokenized query.

from rank_bm25 import BM25Okapi

corpus = [
    "Hello there, this is a document.",
    "This document is about BM25.",
    "Hello, how are you today?",
    "BM25 is a ranking algorithm.",
]

# Tokenize the corpus (essential step)
tokenized_corpus = [doc.lower().split(" ") for doc in corpus]

bm25 = BM25Okapi(tokenized_corpus)

query = "BM25 ranking algorithm"
tokenized_query = query.lower().split(" ")

doc_scores = bm25.get_scores(tokenized_query)
print(f"Document scores: {doc_scores}")

top_n = bm25.get_top_n(tokenized_query, corpus, n=2)
print(f"Top 2 documents: {top_n}")

view raw JSON →