bm25s

Warnings

• breaking Starting with version 0.3.0, `scipy` is no longer a required dependency. The library now uses a pure NumPy-based CSC matrix builder by default. If you relied on SciPy's CSC builder, you must install `scipy` separately and pass `csc_backend="scipy"` to the `BM25()` constructor, or install `bm25s` with the `[indexing]` extra (e.g., `pip install bm25s[indexing]`).
  Fix: Explicitly install `scipy` and pass `csc_backend="scipy"` to `BM25()`, or install with `pip install bm25s[indexing]`.
• breaking The internal import path for the `selection` module changed in version 0.3.0. If you were directly importing `selection` (e.g., `from bm25s import selection`), you should update your imports to `from bm25s import selection_np`.
  Fix: Update import statements from `from bm25s import selection` to `from bm25s import selection_np`.
• gotcha For simpler, beginner-friendly usage and a high-level API that includes all necessary dependencies, the project now recommends using the separate `bm25` package (i.e., `pip install bm25`). Users should be aware of the distinction if seeking a fully batteries-included experience.
  Fix: Consider `pip install bm25` for a higher-level API, or explicitly manage `bm25s` dependencies (like `PyStemmer`) for advanced control.
• gotcha Achieving optimal performance and relevance often requires optional dependencies such as `PyStemmer` for stemming, `numba` for JIT compilation, or `jax[cpu]` for accelerated top-k selection. Without these, performance or search quality might not meet expectations, especially on large datasets.
  Fix: Install recommended optional dependencies like `pip install PyStemmer`, `pip install numba`, or `pip install jax[cpu]` based on your performance and feature needs.

Install

• pip install bm25s Base installation
• pip install bm25s[full] Recommended: with all extra dependencies (stemming, Numba, etc.)
• pip install bm25s[indexing] # for SciPy's CSC builder if preferred pip install jax[cpu] # for JAX-based top-k selection speedup Optional: Specific backends/optimizations

Imports

• BM25

  from bm25s import BM25

• tokenize

  from bm25s import tokenize

• selection

  from bm25s import selection_np

  As of v0.3.0, `selection` was renamed internally to `selection_np` to avoid conflicts and signify its NumPy-based nature. Direct imports should be updated.

Quickstart

This quickstart demonstrates how to initialize BM25S, tokenize a corpus (with optional stemming and stopword removal), index the documents, and perform a query to retrieve the top-k most relevant documents. It showcases the core `BM25` class and `tokenize` utility.

import bm25s
import Stemmer # Ensure 'pip install PyStemmer' is run for this

corpus = [
    "a cat is a feline and likes to purr",
    "a dog is the human's best friend and loves to play",
    "a bird is a beautiful animal that can fly",
    "a fish is a creature that lives in water and swims",
]

# Optional: create a stemmer and tokenizer
# For optimal results, ensure PyStemmer is installed or provide your own tokenizer.
stemmer = Stemmer.Stemmer("english")
tokenized_corpus = bm25s.tokenize(corpus, stemmer=stemmer, stopwords="english")

# Create the BM25 model and index the corpus
retriever = bm25s.BM25(corpus=corpus)
retriever.index(tokenized_corpus)

# Query the corpus and get top-k results
query = "does the fish purr like a cat?"
tokenized_query = bm25s.tokenize(query, stemmer=stemmer, stopwords="english")
results, scores = retriever.retrieve(tokenized_query, k=2)

# Print the ranked results
print(f"Query: '{query}'")
print("Top results:")
for i in range(results.shape[0]):
    doc_id = results[i, 0]
    score = scores[i, 0]
    print(f"  (score: {score:.2f}): {corpus[doc_id]}")