OpenTelemetry Lancedb Instrumentation

0.58.0 · active · verified Fri Apr 10

This library provides OpenTelemetry instrumentation for LanceDB, allowing tracing of client-side calls to the LanceDB vector database. It is part of the OpenLLMetry project, which extends OpenTelemetry for LLM applications. The library maintains a rapid release cadence, with multiple updates per month to keep pace with semantic convention changes and library updates.

Warnings

breaking The OpenLLMetry project, which includes this instrumentation, frequently updates its semantic conventions, especially for GenAI attributes. Existing dashboards or analytical queries relying on specific span attribute names or structures related to LLM/vector DB operations may break or show incorrect data after updates.
Fix: Review the latest OpenTelemetry GenAI Semantic Conventions and OpenLLMetry documentation for updated attribute names. Adapt your monitoring and analysis tools accordingly after upgrading the instrumentation library.
gotcha This instrumentation library only adds hooks to LanceDB. For traces to be generated and exported, a full OpenTelemetry SDK setup (TracerProvider, SpanProcessor, and SpanExporter) must be configured in your application. Without this, no telemetry will be collected or visible.
Fix: Ensure you explicitly configure an OpenTelemetry `TracerProvider` and register a `SpanProcessor` with an appropriate `SpanExporter` (e.g., `ConsoleSpanExporter`, `OTLPSpanExporter`) before initializing the `LancedbInstrumentation`.
gotcha This instrumentation library focuses solely on LanceDB. If you are using other LLM frameworks (e.g., OpenAI, LangChain, LlamaIndex) in conjunction with LanceDB, you will need to install and instrument those libraries separately to get comprehensive end-to-end traces for your LLM application.
Fix: Install and instrument all relevant libraries in your LLM application stack (e.g., `opentelemetry-instrumentation-openai`, `opentelemetry-instrumentation-langchain`) to ensure complete observability.

Install

pip install opentelemetry-instrumentation-lancedb lancedb opentelemetry-sdk Install library and core OpenTelemetry SDK

Imports

LancedbInstrumentation

from opentelemetry.instrumentation.lancedb import LancedbInstrumentation

Quickstart

This example demonstrates how to set up basic OpenTelemetry tracing with a console exporter and then instrument LanceDB. Operations like connecting, creating tables, adding data, and searching will generate traces printed to the console.

import lancedb
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor
from opentelemetry.instrumentation.lancedb import LancedbInstrumentation

# 1. Setup OpenTelemetry Tracer Provider and Exporter
resource = Resource.create({"service.name": "lancedb-app"})
tracer_provider = TracerProvider(resource=resource)
span_processor = SimpleSpanProcessor(ConsoleSpanExporter())
tracer_provider.add_span_processor(span_processor)
trace.set_tracer_provider(tracer_provider)

# 2. Initialize LanceDB Instrumentation
LancedbInstrumentation().instrument()

# 3. Use LanceDB (will now be instrumented)
print("\n--- Performing LanceDB operations ---")
uri = ".lancedb_test"
db = lancedb.connect(uri)

# Create a table
data = [
    {"vector": [0.1, 0.2], "text": "hello"},
    {"vector": [0.3, 0.4], "text": "world"}
]
table = db.create_table("my_table", data=data, mode="overwrite")

# Add more data
table.add([{"vector": [0.5, 0.6], "text": "opentelemetry"}])

# Search data
results = table.search([0.2, 0.3]).limit(1).to_list()
print(f"Search results: {results}")

# Clean up
db.drop_table("my_table")
print("--- LanceDB operations complete ---")

view raw JSON →