OpenInference LiteLLM Instrumentation

Warnings

• gotcha When tracing image generation calls, the instrumentation currently sets the output as a URL attribute rather than rendering the image directly within the trace. Displaying the image requires a UI-side change in the observability tool.
  Fix: This is by design; handle image URLs in your observability tool's UI for visualization.
• gotcha There may be inconsistencies in how output (parsed vs. raw object) is set in span attributes for streamed versus non-streamed LiteLLM calls. While recent updates (v0.1.21) improved full JSON output, manual verification of span attributes for different call types is recommended.
  Fix: Inspect generated spans in your tracing UI to understand the exact format of 'input' and 'output' attributes for streamed and non-streamed calls.
• gotcha The `litellm.responses` function (which is labeled as 'beta' in LiteLLM) is not directly instrumented by `openinference-instrumentation-litellm`.
  Fix: If tracing `litellm.responses` is critical, you may need to add manual OpenTelemetry spans around these calls or await official instrumentation support.
• breaking If you are using LiteLLM version 1.0.0 or higher, be aware that LiteLLM itself introduced breaking changes, including requiring `openai>=1.0.0`, changes to error types (e.g., `openai.InvalidRequestError` to `openai.BadRequestError`), and response objects inheriting from `BaseModel` instead of `OpenAIObject`. While these are LiteLLM's changes, they impact how your application interacts with instrumented LiteLLM calls.
  Fix: Consult the LiteLLM v1.0.0+ migration guide and update your application code to handle the new LiteLLM API surface, ensuring compatibility with the instrumentation.
• gotcha For proper auto-instrumentation, the `LiteLLMInstrumentor().instrument()` call and the OpenTelemetry `TracerProvider` setup must occur *before* any LiteLLM calls are made in your application. Loading instrumentation too late can result in untraced operations.
  Fix: Ensure the instrumentation setup code is executed at the very beginning of your application's lifecycle, typically immediately after importing necessary modules and before any LiteLLM specific code runs. For complex setups, consider using `-r` flag for module loading.

Install

• pip install openinference-instrumentation-litellm litellm opentelemetry-sdk opentelemetry-exporter-otlp Install with core OpenTelemetry dependencies

Imports

• LiteLLMInstrumentor

  from openinference.instrumentation.litellm import LiteLLMInstrumentor

• TracerProvider

  from opentelemetry.sdk.trace import TracerProvider

• SimpleSpanProcessor

  from opentelemetry.sdk.trace.export import SimpleSpanProcessor

• OTLPSpanExporter

  from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

Quickstart

This quickstart demonstrates how to set up OpenInference LiteLLM Instrumentation with a basic OpenTelemetry configuration, making an LLM call via LiteLLM. It initializes a `TracerProvider`, configures an `OTLPSpanExporter` (e.g., for a local Phoenix collector), instruments LiteLLM, and then executes a sample `completion` and `embedding` call. Remember to set your `OPENAI_API_KEY` environment variable.

import os
import litellm
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from openinference.instrumentation.litellm import LiteLLMInstrumentor

# Configure OpenTelemetry Tracer Provider
resource = Resource.create({
    "service.name": "my-litellm-app"
})

tracer_provider = TracerProvider(resource=resource)

# Example: Export traces to a local OpenTelemetry Collector (e.g., Phoenix)
# Ensure a collector is running, e.g., 'python -m phoenix.server.main serve'
OTEL_EXPORTER_OTLP_ENDPOINT = os.environ.get("OTEL_EXPORTER_OTLP_ENDPOINT", "http://127.0.0.1:6006/v1/traces")
tracer_provider.add_span_processor(SimpleSpanProcessor(OTLPSpanExporter(OTEL_EXPORTER_OTLP_ENDPOINT)))

# Set the global tracer provider
from opentelemetry import trace
trace.set_tracer_provider(tracer_provider)

# Instrument LiteLLM
LiteLLMInstrumentor().instrument(tracer_provider=tracer_provider)

# Set LiteLLM API key (e.g., for OpenAI model)
os.environ["OPENAI_API_KEY"] = os.environ.get("OPENAI_API_KEY", "YOUR_OPENAI_API_KEY_HERE")

try:
    print("Making a LiteLLM completion call...")
    completion_response = litellm.completion(
        model="gpt-3.5-turbo",
        messages=[{"content": "What's the capital of France?", "role": "user"}]
    )
    print("Completion received:", completion_response.choices[0].message.content)

    print("\nMaking a LiteLLM embedding call...")
    embedding_response = litellm.embedding(
        model="text-embedding-ada-002",
        input=["Hello, world!"]
    )
    print("Embedding received (first 10 chars):", str(embedding_response.data[0].embedding)[:10] + "...")

except Exception as e:
    print(f"An error occurred: {e}")
    print("Please ensure your API key is set correctly and the model is accessible.")

finally:
    # It's important to shut down the tracer provider to ensure all spans are exported.
    print("\nShutting down tracer provider...")
    tracer_provider.shutdown()
    print("Traces exported.")