OpenTelemetry SageMaker Instrumentation

0.58.0 · active · verified Fri Apr 10

The opentelemetry-instrumentation-sagemaker library provides OpenTelemetry tracing for interactions with Amazon SageMaker, specifically focusing on model invocations made via the Boto3 client. It captures telemetry data related to SageMaker endpoint calls, enabling observability into ML workflows. The library is currently at version 0.58.0 and maintains a rapid release cadence, often releasing weekly or bi-weekly.

Warnings

gotcha By default, the instrumentation logs SageMaker endpoint request bodies and responses as span attributes. This may expose sensitive data. It can also increase trace size significantly.
Fix: Set the environment variable `TRACELOOP_TRACE_CONTENT=false` to disable logging of request and response content. This is highly recommended for production environments dealing with sensitive information.
breaking The `opentelemetry-instrumentation-sagemaker` library, along with other OpenTelemetry GenAI instrumentations, is actively adopting the evolving OpenTelemetry Generative AI Semantic Conventions. This may lead to changes in span names, attribute keys, and overall trace structure between minor versions, especially when new experimental conventions are introduced.
Fix: Review the release notes for your specific version and related OpenTelemetry GenAI projects. If you rely on specific attribute names or trace structures, you may need to update your data processing logic. The `OTEL_SEMCONV_STABILITY_OPT_IN` environment variable (e.g., `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental`) can be used to explicitly opt into the latest experimental conventions or maintain older ones, but careful testing is advised during upgrades.
gotcha This instrumentation specifically targets SageMaker calls made through the Boto3 client. If you require broader instrumentation for other AWS services or general Boto3 operations, you might also need to install and instrument `opentelemetry-instrumentation-boto3` or `opentelemetry-instrumentation-botocore` separately.
Fix: Identify all AWS services you interact with. Install `opentelemetry-instrumentation-boto3` or `opentelemetry-instrumentation-botocore` if you need general AWS SDK tracing beyond SageMaker. Enable each instrumentor explicitly via `MyInstrumentor().instrument()`.
gotcha For traces to be collected and exported, you must have a configured OpenTelemetry `TracerProvider` and a `SpanProcessor` with an appropriate `SpanExporter` (e.g., OTLP, Console). Merely installing the instrumentation package and calling `instrument()` will not result in visible traces without the core SDK setup.
Fix: Always ensure your application includes the setup for `opentelemetry.sdk.trace.TracerProvider`, adds a `SpanProcessor`, and configures an `SpanExporter` that sends data to your observability backend (e.g., OTLP exporter to a collector, or console exporter for local debugging).

Install

pip install opentelemetry-instrumentation-sagemaker opentelemetry-sdk opentelemetry-exporter-otlp Install package and core OTel components

Imports

SageMakerInstrumentor

from opentelemetry.instrumentation.sagemaker import SageMakerInstrumentor

TracerProvider
```
from opentelemetry.sdk.trace import TracerProvider
```
The SDK TracerProvider is used for configuration, while `opentelemetry.trace` provides the API interfaces.

Quickstart

This quickstart demonstrates how to enable OpenTelemetry SageMaker instrumentation and manually configure the SDK to export traces to the console. It uses a mocked boto3 client to simulate an `invoke_endpoint` call without requiring actual AWS credentials, allowing you to see the generated spans. The `TRACELOOP_TRACE_CONTENT` environment variable is set to 'false' to prevent logging of sensitive request/response data.

import os
from unittest.mock import MagicMock

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor
from opentelemetry.instrumentation.sagemaker import SageMakerInstrumentor

# Set up OpenTelemetry SDK for console output
provider = TracerProvider()
processor = SimpleSpanProcessor(ConsoleSpanExporter())
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

# Instrument SageMaker (Boto3 calls)
SageMakerInstrumentor().instrument()

# Disable logging of request/response bodies for privacy (optional but recommended)
os.environ['TRACELOOP_TRACE_CONTENT'] = 'false'

# Mock boto3 client to avoid actual AWS calls
mock_client = MagicMock()
mock_client.invoke_endpoint.return_value = {
    'Body': MagicMock(read=lambda: b'{"predictions": [0.9]}'),
    'ContentType': 'application/json',
    'InvokedProductionVariant': 'variant-1'
}

# Use the instrumented client (even if mocked, instrumentation hooks will be called)
print('Invoking SageMaker endpoint (mocked)...')
response = mock_client.invoke_endpoint(
    EndpointName='my-ml-endpoint',
    ContentType='application/json',
    Body=b'{"instances": [[1,2,3]]}'
)

print(f"Mocked response: {response['Body'].read().decode()}")
print("Check console for OpenTelemetry trace output.")

# To uninstrument (optional)
# SageMakerInstrumentor().uninstrument()

view raw JSON →