Model Hosting Container Standards

Warnings

• gotcha Avoid hardcoding sensitive information like API keys or Hugging Face tokens directly into container images or deployment scripts. Always use environment variables, AWS Secrets Manager, or other secure credential management systems for runtime injection.
  Fix: Use environment variables (e.g., `os.environ.get('HUGGING_FACE_HUB_TOKEN', '')`) or integrate with cloud-native secret management services like AWS Secrets Manager.
• gotcha Failing to set resource limits (CPU/Memory) for containers can lead to resource starvation, instability, and poor performance, especially in multi-container environments on a single instance.
  Fix: Configure appropriate CPU and memory limits for your SageMaker endpoints or Docker containers (e.g., using `ContainerHostResourceLimits` in SageMaker or `--cpus`, `--memory` in Docker).
• gotcha Relying on generic or 'latest' tags for container images in production deployments can introduce instability and make reproducibility difficult due to unexpected upstream changes.
  Fix: Always pin to specific, immutable image tags (e.g., `vllm:0.11.2-sagemaker-v1.2`) for production deployments to ensure consistent behavior and enable rollbacks.
• gotcha Building overly large container images increases deployment times, storage costs, and the attack surface. This is a common issue with custom ML containers.
  Fix: Optimize Dockerfiles by using minimal base images (e.g., `alpine` variants), multi-stage builds, and `.dockerignore` files to exclude unnecessary build artifacts and development dependencies.
• breaking The library explicitly requires Python 3.10 or newer. Deploying or developing with older Python versions (e.g., 3.9 or earlier) will lead to compatibility issues and failures.
  Fix: Ensure your development and deployment environments use Python 3.10 or a newer compatible version.

Install

• pip install model-hosting-container-standards PyPI

Imports

• ModelHandler

  from model_hosting_container_standards.common.handler.model_handler import ModelHandler

  A common component for defining custom model logic within a standardized container.
• FastAPI

  from fastapi import FastAPI

  While FastAPI is a dependency, the toolkit integrates with it for container APIs. Users might import it directly when building custom services using the toolkit's patterns.

Quickstart

This quickstart demonstrates how to deploy a model using Amazon SageMaker, leveraging a container that adheres to the `model-hosting-container-standards`. It configures a SageMaker model with a vLLM-powered container image, setting crucial environment variables for model ID, resource allocation, and optional authentication tokens. This example assumes appropriate AWS credentials and SageMaker execution role are configured in your environment. Note that the toolkit itself is for *building* such containers, and this quickstart shows how to *consume* them on SageMaker.

import boto3
import os

sagemaker_client = boto3.client('sagemaker')

# Replace with your AWS account ID and region
account_id = os.environ.get('AWS_ACCOUNT_ID', '123456789012')
region = os.environ.get('AWS_REGION', 'us-east-1')

model_name = 'my-vllm-standard-model'
execution_role_arn = os.environ.get('SAGEMAKER_EXECUTION_ROLE_ARN', 'arn:aws:iam::123456789012:role/SageMakerExecutionRole')

# Example of using a vLLM container image that adheres to the standards
# This image would typically be found in Amazon ECR Public Gallery or a private ECR repo
# Note: This is an example, use an actual vLLM image URL from AWS ECR Public Gallery.
vllm_image = f"{account_id}.dkr.ecr.{region}.amazonaws.com/vllm:0.11.2-sagemaker-v1.2"

response = sagemaker_client.create_model(
    ModelName=model_name,
    ExecutionRoleArn=execution_role_arn,
    PrimaryContainer={
        'Image': vllm_image,
        'Environment': {
            'SM_VLLM_MODEL': 'meta-llama/Meta-Llama-3-8B-Instruct', # Hugging Face Model ID or S3 path
            'HUGGING_FACE_HUB_TOKEN': os.environ.get('HUGGING_FACE_HUB_TOKEN', ''), # Securely provide token
            'SM_VLLM_MAX_MODEL_LEN': '2048',
            'SM_VLLM_GPU_MEMORY_UTILIZATION': '0.9',
            'SM_VLLM_DTYPE': 'auto',
            'SM_VLLM_TENSOR_PARALLEL_SIZE': '1'
        }
    }
)

print(f"Model creation initiated: {response['ModelArn']}")
# Further steps would involve creating an Endpoint Configuration and an Endpoint