Annoy (Approximate Nearest Neighbors)

Warnings

• gotcha Once the `build()` method is called on an `AnnoyIndex` instance, no more items can be added to that index. Annoy is designed for static, read-only indexes after creation. If you need a mutable index, consider rebuilding or using an alternative library.
  Fix: Plan your data ingestion to add all items before calling `.build()`. If your dataset changes, you must rebuild the entire index.
• gotcha Item IDs must be non-negative integers. Annoy allocates memory for `max(id)+1` items, assuming dense integer IDs from 0 to N-1. Using sparse or very large IDs can lead to excessive memory allocation or unexpected behavior.
  Fix: Map your arbitrary item identifiers to a dense range of non-negative integers (e.g., 0, 1, ..., N-1) before adding them to Annoy.
• gotcha The `n_trees` parameter (during build) affects build time and index size; higher values give better accuracy but larger indexes. The `search_k` parameter (during search) affects search time; higher values give better accuracy but longer search times. You must tune these parameters for your specific accuracy and performance needs.
  Fix: Experiment with different `n_trees` (e.g., 10-1000) during index creation and `search_k` (e.g., `n_trees * 2` or more) during query time to find the optimal trade-off for your dataset and latency requirements.
• deprecated Older versions (prior to 1.17.2) were known to have memory leaks, especially during index building or repeated operations.
  Fix: Upgrade to version 1.17.2 or newer to benefit from memory leak fixes.
• breaking Version 1.16.1 introduced stricter checks, preventing saving an index that hasn't been built or building an index that has already been built.
  Fix: Ensure `build()` is called exactly once before `save()`, and only call `build()` on an index that has not been built yet.
• gotcha Compilation issues have occurred on specific platforms, such as OS X (fixed in 1.17.3) and certain GCC versions with AVX instructions (fixed in 1.16.1). These can prevent successful installation or lead to runtime errors.
  Fix: Ensure you are using the latest stable version of Annoy. If issues persist, check the GitHub issues for platform-specific workarounds or compiler flags.

Install

• pip install annoy Install latest version

Imports

• AnnoyIndex

  from annoy import AnnoyIndex

Quickstart

This example demonstrates how to initialize an Annoy index, add items (vectors), build the index for efficient search, save it to disk, load it back (memory-mapped), and then perform nearest neighbor queries using an item ID or a new vector. The `AnnoyIndex` constructor takes the vector dimension `f` and the distance `metric` (e.g., 'euclidean', 'angular'). The `build` method specifies the number of trees (`n_trees`) and jobs (`n_jobs`).

import os
from annoy import AnnoyIndex
import random

f = 40  # Length of item vector that will be indexed
t = AnnoyIndex(f, 'euclidean')  # or 'angular', 'manhattan', 'hamming', 'dot'

# Add items to the index
for i in range(1000):
    v = [random.gauss(0, 1) for _ in range(f)]
    t.add_item(i, v)

# Build the index with n_trees trees. n_jobs=-1 uses all CPU cores.
t.build(10, n_jobs=-1) 

# Save and load the index
index_path = 'test.ann'
t.save(index_path)

u = AnnoyIndex(f, 'euclidean')
u.load(index_path) # super fast, will just mmap the file

# Query for nearest neighbors
query_item_id = 0
k = 10 # Number of neighbors to retrieve

nearest_neighbors = u.get_nns_by_item(query_item_id, k)
print(f"Nearest neighbors for item {query_item_id}: {nearest_neighbors}")

query_vector = [random.gauss(0, 1) for _ in range(f)]
nearest_neighbors_by_vector = u.get_nns_by_vector(query_vector, k)
print(f"Nearest neighbors for a random vector: {nearest_neighbors_by_vector}")

# Clean up the created index file
if os.path.exists(index_path):
    os.remove(index_path)