IBM COS S3 Transfer Manager

2.16.1 · active · verified Sun Apr 12

The `ibm-cos-sdk-s3transfer` library provides a high-level, multi-part, multi-threaded Amazon S3-like transfer manager optimized for IBM Cloud Object Storage (COS). It handles efficient uploading and downloading of files, especially large objects, managing part sizing and concurrency. It is currently at version 2.16.1 and receives regular updates, typically with minor feature releases every few months and patch releases for bug fixes.

Warnings

gotcha The `ibm-cos-sdk-s3transfer` library is generally a transitive dependency of `ibm-cos-sdk`. For typical usage, you should install `ibm-cos-sdk` and import `TransferManager` from `ibm_cos_sdk.s3.transfer`, not directly from `ibm_cos_sdk_s3transfer`.
Fix: Install `ibm-cos-sdk` and use `from ibm_cos_sdk.s3.transfer import TransferManager`.
gotcha This library requires Python 3.8 or newer. Using older Python versions will result in installation or runtime errors.
Fix: Ensure your Python environment is 3.8 or newer. Upgrade Python if necessary.
gotcha IBM Cloud Object Storage (COS) requires specific configuration parameters that differ from standard AWS S3. Notably, you must provide `ibm_api_key_id`, `ibm_service_instance_id`, and a COS-specific `endpoint_url` when initializing the client.
Fix: Refer to IBM COS documentation for correct endpoint URLs and ensure all required `ibm_` prefixed parameters are passed to `ibm_cos_sdk.client('s3', ...)`.
gotcha When transferring very large files or managing many concurrent transfers, `TransferManager` can consume significant system resources (memory, CPU). Monitor resource usage and adjust `max_concurrency` or `multipart_threshold` parameters if encountering issues.
Fix: For fine-grained control, consider initializing `TransferManager` with custom `transfer_config` options. For example, `TransferConfig(max_concurrency=N)` to limit concurrent parts or `TransferConfig(multipart_threshold=M_bytes)` to adjust the multipart upload threshold.

Install

pip install ibm-cos-sdk Recommended: Install with the full IBM COS SDK
pip install ibm-cos-sdk-s3transfer Install only the transfer manager (less common)

Imports

TransferManager
```
from ibm_cos_sdk.s3.transfer import TransferManager
```
`ibm-cos-sdk-s3transfer` is primarily an internal dependency of `ibm-cos-sdk`. The `TransferManager` object is exposed through the higher-level `ibm_cos_sdk.s3.transfer` module for end-user convenience.
Client
```
import ibm_cos_sdk
```
The COS client object needed for TransferManager is obtained from `ibm_cos_sdk.client('s3', ...)`.

Quickstart

This quickstart demonstrates how to initialize the IBM COS client and then use the `TransferManager` to upload and download a file. It expects IBM COS credentials and bucket name to be set as environment variables.

import os
import ibm_cos_sdk
from ibm_cos_sdk.s3.transfer import TransferManager

# Configure IBM COS credentials and endpoint
# Ensure these environment variables are set:
# IBM_COS_ENDPOINT, IBM_COS_API_KEY_ID, IBM_COS_SERVICE_INSTANCE_ID, IBM_COS_BUCKET

ENDPOINT = os.environ.get('IBM_COS_ENDPOINT', 'https://s3.us.cloud-object-storage.appdomain.cloud')
API_KEY_ID = os.environ.get('IBM_COS_API_KEY_ID', '')
SERVICE_INSTANCE_ID = os.environ.get('IBM_COS_SERVICE_INSTANCE_ID', '')
BUCKET_NAME = os.environ.get('IBM_COS_BUCKET', 'my-test-bucket')

if not all([API_KEY_ID, SERVICE_INSTANCE_ID, BUCKET_NAME]):
    print("Error: IBM COS environment variables not set. Please set IBM_COS_API_KEY_ID, IBM_COS_SERVICE_INSTANCE_ID, IBM_COS_BUCKET.")
    exit(1)

# Initialize COS client
cos_client = ibm_cos_sdk.client(
    's3',
    ibm_api_key_id=API_KEY_ID,
    ibm_service_instance_id=SERVICE_INSTANCE_ID,
    config=ibm_cos_sdk.config.Config(signature_version='oauth'),
    endpoint_url=ENDPOINT
)

# Create a TransferManager instance
transfer_manager = TransferManager(cos_client)

# --- Example: Upload a file ---
local_file_path = 'local_file.txt'
key_name = 'remote_object.txt'

# Create a dummy local file for upload
with open(local_file_path, 'w') as f:
    f.write('Hello, IBM COS Transfer Manager!')

print(f"Uploading {local_file_path} to {BUCKET_NAME}/{key_name}...")

try:
    future = transfer_manager.upload(local_file_path, BUCKET_NAME, key_name)
    future.result()  # Wait for the transfer to complete
    print(f"Successfully uploaded {key_name}")
except Exception as e:
    print(f"Upload failed: {e}")

# --- Example: Download a file ---
download_path = 'downloaded_file.txt'

print(f"Downloading {BUCKET_NAME}/{key_name} to {download_path}...")

try:
    future = transfer_manager.download(BUCKET_NAME, key_name, download_path)
    future.result() # Wait for the transfer to complete
    print(f"Successfully downloaded {key_name} to {download_path}")
except Exception as e:
    print(f"Download failed: {e}")

# Clean up local files (optional)
os.remove(local_file_path)
os.remove(download_path)

# You can also close the transfer manager (optional)
transfer_manager.shutdown()

view raw JSON →