Hnswlib

Warnings

• breaking Indices saved with very old versions (prior to v0.3.4) are not supported and cannot be loaded with newer `hnswlib` versions.
  Fix: Rebuild indices with a supported `hnswlib` version. Consider exporting data and re-importing if migration is critical.
• breaking Saving and loading of large pickled indices (greater than 4GB) in versions prior to 0.6.2 could lead to data corruption.
  Fix: Upgrade to `hnswlib` v0.6.2 or later to prevent corruption of large pickled indices. Rebuild any potentially corrupted indices.
• breaking Indices built with AVX512 or AVX optimizations (enabled during compilation) in `hnswlib` v0.6.1 and later may not be backwards-compatible with older SSE or non-AVX512 architectures. This can cause issues when moving indices between machines with different CPU capabilities.
  Fix: Ensure that the `hnswlib` library is compiled and indices are used on machines with compatible CPU architectures and instruction sets. Recompile without specific AVX flags if maximum portability is needed, or rebuild indices on the target architecture.
• gotcha The `ef` parameter, which controls the query-time accuracy/speed trade-off, is *not* saved as part of the index. It must be manually set after loading a saved index.
  Fix: Always call `index.set_ef(value)` after loading an index to configure the desired query performance.
• breaking In `hnswlib` v0.8.0, statistic aggregation was removed by default for multi-threaded search to improve speed. Users who relied on this feature might observe changes in behavior or require explicit configuration if it's still needed.
  Fix: Review multi-threaded search logic; if statistic aggregation was implicitly relied upon, evaluate impact. Check release notes for explicit alternatives if available.
• gotcha When performing brute-force searches with filters, versions prior to 0.9.0 (currently on GitHub, not yet PyPI stable) contained bugs that could lead to incorrect results or missing normalization checks. Additionally, searching for `k` elements when fewer than `k` are available now explicitly throws an exception.
  Fix: Upgrade to `hnswlib` v0.9.0 or later once released on PyPI for correct filter behavior and robust error handling when `k` exceeds available elements.

Install

• pip install hnswlib Install from PyPI

Imports

• Index

  import hnswlib
  import numpy as np
  
  index = hnswlib.Index(space='l2', dim=128)

Quickstart

This quickstart demonstrates how to create an HNSW index, initialize it with parameters, add vector data, perform a k-nearest neighbor query, and then save and load the index. It highlights the importance of re-setting the `ef` parameter after loading the index.

import hnswlib
import numpy as np
import os

# Define data parameters
dim = 128
num_elements = 10000

# Generate random data
data = np.float32(np.random.random((num_elements, dim)))
data_labels = np.arange(num_elements)

# Initialize the HNSW index
# Possible space options: 'l2', 'ip' (inner product), 'cosine'
space_name = 'l2' # Euclidean distance
index = hnswlib.Index(space=space_name, dim=dim)

# Set index parameters BEFORE adding data
# max_elements: current capacity
# ef_construction: accuracy vs. construction speed trade-off
# M: number of bi-directional links per data point
index.init_index(max_elements=num_elements, ef_construction=200, M=16)

# Add items to the index
index.add_items(data, data_labels)

# Set query time accuracy/speed trade-off
# Note: This parameter is NOT saved with the index and must be set after loading.
index.set_ef(50)

# Generate a query vector
query_vector = np.float32(np.random.random((1, dim)))

# Perform a k-nearest neighbor query
k = 5
labels, distances = index.knn_query(query_vector, k=k)

print(f"Query vector: {query_vector[0][:5]}...")
print(f"Nearest neighbor labels: {labels[0]}")
print(f"Distances to neighbors: {distances[0]}")

# Example of saving and loading the index
index_path = 'my_hnsw_index.bin'
index.save_index(index_path)

loaded_index = hnswlib.Index(space=space_name, dim=dim)
loaded_index.load_index(index_path)
loaded_index.set_ef(50) # Re-set ef after loading

loaded_labels, loaded_distances = loaded_index.knn_query(query_vector, k=k)
print(f"Loaded index nearest neighbor labels: {loaded_labels[0]}")

os.remove(index_path)