Tantivy Python Bindings

Warnings

• gotcha To install `tantivy` from source (if no pre-compiled wheel is available for your system), you must have Rust installed and configured. This is a common requirement for Python libraries with Rust bindings.
  Fix: Install Rust via `rustup` before attempting `pip install tantivy`.
• breaking Version 0.25.0 introduced a breaking API change by removing index sorting. Users relying on this feature will need to adjust their indexing and search strategies.
  Fix: Review the Tantivy Rust documentation (or tantivy-py changelog) for alternatives or re-architect solutions that previously relied on index sorting.
• gotcha Tantivy treats document data as immutable. To 'edit' a document, you must delete the existing document (by its `DocAddress` or a specific term query) and then reindex the updated version.
  Fix: Implement a delete-and-reindex strategy for document updates.
• gotcha Only one `IndexWriter` can be active at a time for a given index. While the `IndexWriter` itself is multithreaded, concurrent attempts to create multiple writers will fail.
  Fix: Manage access to the `IndexWriter` to ensure singularity, typically by having a single process or thread manage all write operations to an index, or using a locking mechanism in multi-process/thread scenarios.
• gotcha Search operations return a list of `(score, DocAddress)` tuples. To retrieve the actual document content, you must use the `DocAddress` with a `Searcher`'s `doc()` method, rather than receiving the document directly in search results.
  Fix: Always use `searcher.doc(doc_address)` to fetch the stored fields of a document after a search.
• gotcha For incremental indexing and efficient document deletion, the field used to identify documents for deletion (e.g., a unique ID) must be an integer field, set to `indexed=True` and `fast=True` in the schema.
  Fix: Ensure your schema defines the unique ID field with `tantivy.SchemaBuilder.add_integer_field("your_id_field", stored=True, indexed=True, fast=True)`.

Install

• pip install tantivy Install with pip

Imports

• tantivy

  import tantivy

• SchemaBuilder

  from tantivy import SchemaBuilder

  Most core classes are directly under the 'tantivy' top-level module.

Quickstart

This quickstart demonstrates how to define a schema, create an in-memory Tantivy index, add documents to it, and perform a basic search. It also shows how to retrieve the full document content from search hits.

import tantivy
import os

# 1. Declare the schema
schema_builder = tantivy.SchemaBuilder()
schema_builder.add_text_field("title", stored=True, tokenizer_name="default")
schema_builder.add_text_field("body", stored=True, tokenizer_name="default")
schema_builder.add_integer_field("doc_id", stored=True, indexed=True)
schema = schema_builder.build()

# 2. Create an in-memory index (for persistent, specify a path)
# To use a persistent index, use: index = tantivy.Index(schema, path="/tmp/my_index")
index = tantivy.Index(schema)

# 3. Get an index writer and add documents
writer = index.writer(50_000_000) # 50MB memory arena
writer.add_document(tantivy.Document(title=["The Old Man and the Sea"], body=["He was an old man who fished alone in a skiff."], doc_id=[1]))
writer.add_document(tantivy.Document(title=["The Great Gatsby"], body=["In my younger and more vulnerable years my father gave me some advice."], doc_id=[2]))
writer.commit()

# 4. Get a reader and searcher
index.reload()
reader = index.reader()
searcher = reader.searcher()

# 5. Build and execute a query
query_parser = tantivy.QueryParser(schema, default_fields=["title", "body"])
query = query_parser.parse_query("old man")
hits = searcher.search(query, 10)

# 6. Retrieve documents
print("Search results:")
for score, doc_address in hits:
    retrieved_doc = searcher.doc(doc_address)
    print(f"  Score: {score:.2f}, Doc ID: {retrieved_doc['doc_id'][0]}, Title: {retrieved_doc['title'][0]}")

# Example of retrieving a non-existent field (will be empty list)
missing_field = retrieved_doc.get('non_existent_field')
print(f"  Non-existent field for last doc: {missing_field}") # Expected: []