Multi-Storage Client (MSC)

Warnings

• breaking Python 3.9 support was dropped in version 0.43.0. Users on Python 3.9 must upgrade to Python 3.10 or newer.
  Fix: Upgrade your Python environment to version 3.10 or later. The library requires Python <4, >=3.10.
• gotcha The Multi-Storage Client heavily relies on external YAML configuration files for defining storage profiles and credentials. Direct in-code configuration for providers/credentials is not the primary pattern.
  Fix: Define your storage provider profiles (including credentials and options) in a YAML file and set the `MSC_CONFIG` environment variable to its path.
• gotcha To support specific cloud storage providers (e.g., AWS S3, Google Cloud Storage), you must install the corresponding optional dependencies using `pip install "multi-storage-client[extra_name]"`. The base `multi-storage-client` package only supports POSIX file systems by default.
  Fix: Install the required extras for your cloud provider, e.g., `pip install "multi-storage-client[boto3]"` for AWS S3.
• gotcha Attempting to import `multistorageclient.instrumentation.auth` without installing the `[vault]` extra will result in an `ImportError`.
  Fix: If you intend to use `vault` integration, install it via `pip install "multi-storage-client[vault]"`. Otherwise, avoid importing the instrumentation.auth module if the extra is not installed.
• deprecated The `cryptography` package was incorrectly made a core requirement in version 0.45.0. This was corrected in a subsequent release (0.45.1) to restore it as an optional dependency, reducing the default install footprint for users not needing its features.
  Fix: Upgrade to version 0.45.1 or newer to ensure `cryptography` is an optional dependency. If you require `cryptography` features, ensure you install the relevant extra (e.g., `[vault]`).

Install

• pip install multi-storage-client Base client (POSIX file systems)
• pip install "multi-storage-client[boto3]" For AWS S3
• pip install "multi-storage-client[google-cloud-storage]" For Google Cloud Storage (GCS)
• pip install "multi-storage-client[azure-storage-blob]" For Azure Blob Storage

Imports

• StorageClient

  from multistorageclient.client import StorageClient

• msc_path

  from multistorageclient.path import msc_path

Quickstart

This quickstart demonstrates how to initialize the `StorageClient` using a YAML configuration file and perform basic file operations (write, read, list) with a local POSIX storage profile. Authentication for cloud providers is typically managed within the YAML profiles or via environment variables, avoiding hardcoding in Python code.

import os
import yaml
from multistorageclient.client import StorageClient

# Create a dummy MSC configuration file for local testing
config_content = '''
profiles:
  local-files:
    storage_provider:
      type: posix
      options:
        base_path: ./msc_data
'''

# Create a directory for local files
os.makedirs('./msc_data', exist_ok=True)

config_path = './msc_config.yaml'
with open(config_path, 'w') as f:
    f.write(config_content)

# Set the MSC_CONFIG environment variable to point to the config file
os.environ['MSC_CONFIG'] = config_path

try:
    # Instantiate the client
    client = StorageClient()

    # Example: Write a file using the 'local-files' profile
    local_file_path = 'msc://local-files/example.txt'
    with client.open(local_file_path, 'w') as f:
        f.write('Hello from Multi-Storage Client!')
    print(f"Content written to {local_file_path}")

    # Example: Read the file
    with client.open(local_file_path, 'r') as f:
        content = f.read()
    print(f"Content read from {local_file_path}: {content}")

    # Example: List objects in the 'local-files' profile
    objects = client.list_objects('msc://local-files/')
    print("Objects in 'msc://local-files/':")
    for obj in objects:
        print(f"- {obj.uri}")

finally:
    # Clean up the dummy config file and data directory
    if os.path.exists(config_path):
        os.remove(config_path)
    if os.path.exists('./msc_data/example.txt'):
        os.remove('./msc_data/example.txt')
    if os.path.exists('./msc_data'):
        os.rmdir('./msc_data')
    del os.environ['MSC_CONFIG']