Run:ai Model Streamer

Warnings

• breaking The C++ backend of the streamer requires specific system libraries: `libcurl4` and `libssl1.1_1`. Without these, the Python SDK will not function correctly, leading to runtime errors during model streaming. This is a common installation footgun, especially in minimal container environments.
  Fix: Ensure `libcurl4` and `libssl1.1_1` are installed on your system or within your container image. For Debian-based systems, this typically involves `sudo apt-get install libcurl4-openssl-dev libssl-dev` (or similar packages matching the required versions).
• gotcha When streaming from cloud object storage (S3, GCS, Azure Blob Storage), specific `runai-model-streamer-*` backend packages (e.g., `runai-model-streamer-gcs`) must be installed in addition to the core `runai-model-streamer` package. Furthermore, proper authentication credentials must be configured via environment variables or service account files for the SDK to access the storage buckets.
  Fix: Install the relevant backend package (`pip install runai-model-streamer-[gcs|s3|azure]`). Configure environment variables such as `GOOGLE_APPLICATION_CREDENTIALS` (for GCS), `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` (for S3), or `AZURE_CLIENT_ID`, `AZURE_TENANT_ID`, `AZURE_CLIENT_SECRET` (for Azure). Refer to the documentation for specific authentication methods.
• deprecated Older versions of `runai-model-streamer` used with vLLM, particularly with `tensor-parallel-size > 1`, exhibited pickling errors or issues with distributed streaming across multiple GPUs. This primarily affected distributed loading.
  Fix: Upgrade to `runai-model-streamer` version 0.15.x or later. Ensure `vllm[runai]` is updated. When using `vllm serve`, include `--model-loader-extra-config '{"distributed":true}'` for optimal distributed loading from object storage.
• gotcha The `Run:ai Model Streamer` is primarily optimized for the `SafeTensors` file format, which enables efficient zero-copy loading directly from storage. While it may handle other formats, performance benefits are most pronounced with `SafeTensors`.
  Fix: Store your AI model weights in the `SafeTensors` format for maximum performance. Tools and libraries like Hugging Face's `safetensors` facilitate saving models in this format.
• gotcha Setting environment variables like `RUNAI_STREAMER_CONCURRENCY` and `RUNAI_STREAMER_MEMORY_LIMIT` can significantly impact performance and resource consumption. Incorrect tuning can lead to suboptimal loading times or out-of-memory issues.
  Fix: Carefully tune `RUNAI_STREAMER_CONCURRENCY` (number of OS threads for reading) and `RUNAI_STREAMER_MEMORY_LIMIT` (CPU buffer size) based on your specific model size, available CPU memory, and network bandwidth to object storage. Refer to the official documentation for guidance on these tunable parameters and their impact on performance.

Install

• pip install runai-model-streamer Core library
• pip install runai-model-streamer-gcs Google Cloud Storage backend
• pip install runai-model-streamer-s3 AWS S3 compatible backend
• pip install runai-model-streamer-azure Azure Blob Storage backend
• pip install vllm[runai] For vLLM integration

Imports

• SafetensorsStreamer

  from runai_model_streamer import SafetensorsStreamer

Quickstart

This quickstart demonstrates how to use `SafetensorsStreamer` to initiate streaming of a model. It creates a dummy `safetensors` file for a runnable example. For actual use, `file_path` should point to your model. When working with cloud storage (S3, GCS, Azure), ensure the respective `runai-model-streamer-*` package is installed and authentication environment variables (e.g., `GOOGLE_APPLICATION_CREDENTIALS` for GCS, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` for S3, `AZURE_CLIENT_ID` for Azure) are correctly configured. The `stream_file` method starts the streaming process, and `get_tensors()` can then be used to retrieve tensors.

import os
from runai_model_streamer import SafetensorsStreamer

# This is a placeholder for a safetensors file. In a real scenario,
# 'model.safetensors' would be a path to your model file.
# For a runnable example, you might create a dummy file or adapt to a real path.
# If streaming from cloud storage, ensure appropriate backend package is installed
# and environment variables for authentication are set (e.g., GOOGLE_APPLICATION_CREDENTIALS for GCS).
# For local testing, ensure 'model.safetensors' exists or is mocked.

# Example of creating a dummy safetensors file for local quickstart demonstration
try:
    from safetensors.torch import save_file
    import torch
    dummy_tensor = {'tensor_key': torch.randn(10, 10)}
    save_file(dummy_tensor, 'model.safetensors')
    file_path = "model.safetensors"

    print(f"Attempting to stream from: {file_path}")

    with SafetensorsStreamer() as streamer:
        streamer.stream_file(file_path)
        print("Successfully started streaming.")
        # In a real scenario, you would then iterate and process tensors:
        # for name, tensor in streamer.get_tensors():
        #    gpu_tensor = tensor.to('cuda:0') # Or another accelerator
        #    print(f"Streamed tensor: {name}, shape: {gpu_tensor.shape}")

    print("Streamer context closed.")
except ImportError:
    print("To run this quickstart with a dummy file, install 'safetensors' and 'torch':")
    print("pip install safetensors torch")
    print("Alternatively, replace 'model.safetensors' with an actual path to your model file.")
except Exception as e:
    print(f"An error occurred during quickstart: {e}")
    print("Ensure the file_path is correct and necessary system libraries (libcurl4, libssl1.1_1) are installed.")
    print("If streaming from cloud storage, verify environment variables for authentication are set.")