{"id":2645,"library":"opentelemetry-instrumentation-sagemaker","title":"OpenTelemetry SageMaker Instrumentation","description":"The opentelemetry-instrumentation-sagemaker library provides OpenTelemetry tracing for interactions with Amazon SageMaker, specifically focusing on model invocations made via the Boto3 client. It captures telemetry data related to SageMaker endpoint calls, enabling observability into ML workflows. The library is currently at version 0.58.0 and maintains a rapid release cadence, often releasing weekly or bi-weekly.","status":"active","version":"0.58.0","language":"en","source_language":"en","source_url":"https://github.com/traceloop/openllmetry/tree/main/packages/opentelemetry-instrumentation-sagemaker","tags":["opentelemetry","observability","sagemaker","aws","instrumentation","tracing","ml","generative-ai"],"install":[{"cmd":"pip install opentelemetry-instrumentation-sagemaker opentelemetry-sdk opentelemetry-exporter-otlp","lang":"bash","label":"Install package and core OTel components"}],"dependencies":[{"reason":"Core OpenTelemetry API, essential for any instrumentation.","package":"opentelemetry-api"},{"reason":"Core OpenTelemetry SDK, required for trace and metric providers.","package":"opentelemetry-sdk"},{"reason":"The underlying AWS SDK for Python that this instrumentation targets for SageMaker interactions.","package":"boto3","optional":true}],"imports":[{"symbol":"SageMakerInstrumentor","correct":"from opentelemetry.instrumentation.sagemaker import SageMakerInstrumentor"},{"note":"The SDK TracerProvider is used for configuration, while `opentelemetry.trace` provides the API interfaces.","wrong":"from opentelemetry.trace import TracerProvider","symbol":"TracerProvider","correct":"from opentelemetry.sdk.trace import TracerProvider"}],"quickstart":{"code":"import os\nfrom unittest.mock import MagicMock\n\nfrom opentelemetry import trace\nfrom opentelemetry.sdk.trace import TracerProvider\nfrom opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor\nfrom opentelemetry.instrumentation.sagemaker import SageMakerInstrumentor\n\n# Set up OpenTelemetry SDK for console output\nprovider = TracerProvider()\nprocessor = SimpleSpanProcessor(ConsoleSpanExporter())\nprovider.add_span_processor(processor)\ntrace.set_tracer_provider(provider)\n\n# Instrument SageMaker (Boto3 calls)\nSageMakerInstrumentor().instrument()\n\n# Disable logging of request/response bodies for privacy (optional but recommended)\nos.environ['TRACELOOP_TRACE_CONTENT'] = 'false'\n\n# Mock boto3 client to avoid actual AWS calls\nmock_client = MagicMock()\nmock_client.invoke_endpoint.return_value = {\n    'Body': MagicMock(read=lambda: b'{\"predictions\": [0.9]}'),\n    'ContentType': 'application/json',\n    'InvokedProductionVariant': 'variant-1'\n}\n\n# Use the instrumented client (even if mocked, instrumentation hooks will be called)\nprint('Invoking SageMaker endpoint (mocked)...')\nresponse = mock_client.invoke_endpoint(\n    EndpointName='my-ml-endpoint',\n    ContentType='application/json',\n    Body=b'{\"instances\": [[1,2,3]]}'\n)\n\nprint(f\"Mocked response: {response['Body'].read().decode()}\")\nprint(\"Check console for OpenTelemetry trace output.\")\n\n# To uninstrument (optional)\n# SageMakerInstrumentor().uninstrument()","lang":"python","description":"This quickstart demonstrates how to enable OpenTelemetry SageMaker instrumentation and manually configure the SDK to export traces to the console. It uses a mocked boto3 client to simulate an `invoke_endpoint` call without requiring actual AWS credentials, allowing you to see the generated spans. The `TRACELOOP_TRACE_CONTENT` environment variable is set to 'false' to prevent logging of sensitive request/response data."},"warnings":[{"fix":"Set the environment variable `TRACELOOP_TRACE_CONTENT=false` to disable logging of request and response content. This is highly recommended for production environments dealing with sensitive information.","message":"By default, the instrumentation logs SageMaker endpoint request bodies and responses as span attributes. This may expose sensitive data. It can also increase trace size significantly.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Review the release notes for your specific version and related OpenTelemetry GenAI projects. If you rely on specific attribute names or trace structures, you may need to update your data processing logic. The `OTEL_SEMCONV_STABILITY_OPT_IN` environment variable (e.g., `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental`) can be used to explicitly opt into the latest experimental conventions or maintain older ones, but careful testing is advised during upgrades.","message":"The `opentelemetry-instrumentation-sagemaker` library, along with other OpenTelemetry GenAI instrumentations, is actively adopting the evolving OpenTelemetry Generative AI Semantic Conventions. This may lead to changes in span names, attribute keys, and overall trace structure between minor versions, especially when new experimental conventions are introduced.","severity":"breaking","affected_versions":"0.53.0 and newer (especially impacting versions adopting 'GenAI semantic conventions' changes)"},{"fix":"Identify all AWS services you interact with. Install `opentelemetry-instrumentation-boto3` or `opentelemetry-instrumentation-botocore` if you need general AWS SDK tracing beyond SageMaker. Enable each instrumentor explicitly via `MyInstrumentor().instrument()`.","message":"This instrumentation specifically targets SageMaker calls made through the Boto3 client. If you require broader instrumentation for other AWS services or general Boto3 operations, you might also need to install and instrument `opentelemetry-instrumentation-boto3` or `opentelemetry-instrumentation-botocore` separately.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Always ensure your application includes the setup for `opentelemetry.sdk.trace.TracerProvider`, adds a `SpanProcessor`, and configures an `SpanExporter` that sends data to your observability backend (e.g., OTLP exporter to a collector, or console exporter for local debugging).","message":"For traces to be collected and exported, you must have a configured OpenTelemetry `TracerProvider` and a `SpanProcessor` with an appropriate `SpanExporter` (e.g., OTLP, Console). Merely installing the instrumentation package and calling `instrument()` will not result in visible traces without the core SDK setup.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-04-10T00:00:00.000Z","next_check":"2026-07-09T00:00:00.000Z"}