txtai

Warnings

• breaking Two packages on PyPI: 'txtai' (full library) and 'txtai.py' (thin API client). They have different APIs. 'pip install txtai.py' installs a client that connects to a remote txtai server — not the local library.
  Fix: For local use: pip install txtai. For connecting to a remote txtai API server: pip install txtai.py
• breaking index() replaces the entire index. Calling it twice means the first index is gone. LLMs commonly generate code that calls index() multiple times to 'add' documents.
  Fix: Use upsert() to add/update documents without full rebuild. Use index() only for initial load or full re-index.
• breaking search() returns (id, score) tuples by default — not document text. Accessing result['text'] raises KeyError without content=True enabled.
  Fix: Enable Embeddings(content=True) to store and retrieve text from search results.
• gotcha SQL queries and content retrieval require content=True at index creation time. Cannot be enabled after index is built without re-indexing.
  Fix: Always set content=True if you need SQL queries, text retrieval, or metadata filtering.
• gotcha Base 'pip install txtai' has minimal deps. Most useful features (pipelines, LLM, API server) require extras: txtai[pipeline-text], txtai[api], txtai[all].
  Fix: For RAG/LLM workflows: pip install txtai[all]. For just semantic search: pip install txtai[similarity].
• gotcha Default model downloads from Hugging Face Hub on first use — requires internet access and ~100MB download. Fails in air-gapped environments.
  Fix: Pre-download model: embeddings = Embeddings(path='/local/model/path'). Or set HF_HUB_OFFLINE=1 with a cached model.
• gotcha Agents (added in v8) are built on smolagents framework — requires pip install txtai[agent]. Earlier versions used transformers agents which had different API.
  Fix: pip install txtai[agent] for agent support.

Install

• pip install txtai Python (full local library — minimal deps)
• pip install txtai[all] Python (all optional deps — pipelines, API, etc.)
• pip install txtai[api] Python (with FastAPI server)
• pip install txtai.py Python (thin client for remote txtai API server)

Imports

• Embeddings (basic semantic search)

  from txtai import Embeddings
  
  # Default model (all-MiniLM-L6-v2)
  embeddings = Embeddings()
  
  # Or specify model explicitly
  embeddings = Embeddings(path='sentence-transformers/all-MiniLM-L6-v2')
  
  # index() — builds NEW index, overwrites existing
  embeddings.index(['Correct answer', 'Wrong answer', 'Maybe'])
  
  # search returns list of (id, score) tuples
  results = embeddings.search('positive', 1)
  print(results)  # [(0, 0.298)] — id=0 is 'Correct answer'

  index() rebuilds the entire index from scratch — calling it again wipes previous data. Use upsert() to add/update without full rebuild. search() returns (id, score) tuples by default, not text.
• Embeddings with content storage

  from txtai import Embeddings
  
  # Enable content storage to retrieve text from search results
  embeddings = Embeddings(content=True)
  
  # Index with dict documents
  embeddings.index([
      {'id': 0, 'text': 'Python is a programming language'},
      {'id': 1, 'text': 'JavaScript runs in browsers'},
      {'id': 2, 'text': 'Rust is fast and safe'},
  ])
  
  # Now search returns dicts with text
  results = embeddings.search('compiled language', 1)
  print(results[0]['text'])  # 'Rust is fast and safe'
  
  # Can also use SQL
  results = embeddings.search(
      "SELECT text, score FROM txtai WHERE similar('web language') LIMIT 1"
  )

  Without content=True, search returns (id, score) tuples only. Enable content=True to store and retrieve document text. SQL queries also require content=True.

Quickstart

txtai Embeddings with content storage, search, upsert, save/load.

# pip install txtai
from txtai import Embeddings

# Create embeddings with content storage
embeddings = Embeddings(
    path='sentence-transformers/all-MiniLM-L6-v2',
    content=True
)

# Index documents
embeddings.index([
    {'id': 0, 'text': 'Python is a programming language created by Guido'},
    {'id': 1, 'text': 'JavaScript is used for web development'},
    {'id': 2, 'text': 'Rust provides memory safety without garbage collection'},
    {'id': 3, 'text': 'Go is designed for cloud infrastructure'},
])

# Semantic search — returns dicts with text
results = embeddings.search('systems programming language', 2)
for r in results:
    print(r['text'], r['score'])

# Upsert — add without rebuilding
embeddings.upsert([{'id': 4, 'text': 'TypeScript adds types to JavaScript'}])

# Save and load
embeddings.save('/tmp/myindex')
embeddings.load('/tmp/myindex')