{"id":4570,"library":"ibm-cos-sdk-s3transfer","title":"IBM COS S3 Transfer Manager","description":"The `ibm-cos-sdk-s3transfer` library provides a high-level, multi-part, multi-threaded Amazon S3-like transfer manager optimized for IBM Cloud Object Storage (COS). It handles efficient uploading and downloading of files, especially large objects, managing part sizing and concurrency. It is currently at version 2.16.1 and receives regular updates, typically with minor feature releases every few months and patch releases for bug fixes.","status":"active","version":"2.16.1","language":"en","source_language":"en","source_url":"https://github.com/IBM/ibm-cos-sdk-python-s3transfer","tags":["IBM Cloud","COS","S3","Object Storage","File Transfer","AWS S3 Compatible"],"install":[{"cmd":"pip install ibm-cos-sdk","lang":"bash","label":"Recommended: Install with the full IBM COS SDK"},{"cmd":"pip install ibm-cos-sdk-s3transfer","lang":"bash","label":"Install only the transfer manager (less common)"}],"dependencies":[{"reason":"Provides the IBM COS client and exposes the TransferManager for direct use. This package is typically a transitive dependency of `ibm-cos-sdk`.","package":"ibm-cos-sdk","optional":false},{"reason":"This library is a fork and extension of the upstream `s3transfer` library from `boto3`.","package":"s3transfer","optional":false}],"imports":[{"note":"`ibm-cos-sdk-s3transfer` is primarily an internal dependency of `ibm-cos-sdk`. The `TransferManager` object is exposed through the higher-level `ibm_cos_sdk.s3.transfer` module for end-user convenience.","wrong":"from ibm_cos_sdk_s3transfer.manager import TransferManager","symbol":"TransferManager","correct":"from ibm_cos_sdk.s3.transfer import TransferManager"},{"note":"The COS client object needed for TransferManager is obtained from `ibm_cos_sdk.client('s3', ...)`.","symbol":"Client","correct":"import ibm_cos_sdk"}],"quickstart":{"code":"import os\nimport ibm_cos_sdk\nfrom ibm_cos_sdk.s3.transfer import TransferManager\n\n# Configure IBM COS credentials and endpoint\n# Ensure these environment variables are set:\n# IBM_COS_ENDPOINT, IBM_COS_API_KEY_ID, IBM_COS_SERVICE_INSTANCE_ID, IBM_COS_BUCKET\n\nENDPOINT = os.environ.get('IBM_COS_ENDPOINT', 'https://s3.us.cloud-object-storage.appdomain.cloud')\nAPI_KEY_ID = os.environ.get('IBM_COS_API_KEY_ID', '')\nSERVICE_INSTANCE_ID = os.environ.get('IBM_COS_SERVICE_INSTANCE_ID', '')\nBUCKET_NAME = os.environ.get('IBM_COS_BUCKET', 'my-test-bucket')\n\nif not all([API_KEY_ID, SERVICE_INSTANCE_ID, BUCKET_NAME]):\n    print(\"Error: IBM COS environment variables not set. Please set IBM_COS_API_KEY_ID, IBM_COS_SERVICE_INSTANCE_ID, IBM_COS_BUCKET.\")\n    exit(1)\n\n# Initialize COS client\ncos_client = ibm_cos_sdk.client(\n    's3',\n    ibm_api_key_id=API_KEY_ID,\n    ibm_service_instance_id=SERVICE_INSTANCE_ID,\n    config=ibm_cos_sdk.config.Config(signature_version='oauth'),\n    endpoint_url=ENDPOINT\n)\n\n# Create a TransferManager instance\ntransfer_manager = TransferManager(cos_client)\n\n# --- Example: Upload a file ---\nlocal_file_path = 'local_file.txt'\nkey_name = 'remote_object.txt'\n\n# Create a dummy local file for upload\nwith open(local_file_path, 'w') as f:\n    f.write('Hello, IBM COS Transfer Manager!')\n\nprint(f\"Uploading {local_file_path} to {BUCKET_NAME}/{key_name}...\")\n\ntry:\n    future = transfer_manager.upload(local_file_path, BUCKET_NAME, key_name)\n    future.result()  # Wait for the transfer to complete\n    print(f\"Successfully uploaded {key_name}\")\nexcept Exception as e:\n    print(f\"Upload failed: {e}\")\n\n# --- Example: Download a file ---\ndownload_path = 'downloaded_file.txt'\n\nprint(f\"Downloading {BUCKET_NAME}/{key_name} to {download_path}...\")\n\ntry:\n    future = transfer_manager.download(BUCKET_NAME, key_name, download_path)\n    future.result() # Wait for the transfer to complete\n    print(f\"Successfully downloaded {key_name} to {download_path}\")\nexcept Exception as e:\n    print(f\"Download failed: {e}\")\n\n# Clean up local files (optional)\nos.remove(local_file_path)\nos.remove(download_path)\n\n# You can also close the transfer manager (optional)\ntransfer_manager.shutdown()\n","lang":"python","description":"This quickstart demonstrates how to initialize the IBM COS client and then use the `TransferManager` to upload and download a file. It expects IBM COS credentials and bucket name to be set as environment variables."},"warnings":[{"fix":"Install `ibm-cos-sdk` and use `from ibm_cos_sdk.s3.transfer import TransferManager`.","message":"The `ibm-cos-sdk-s3transfer` library is generally a transitive dependency of `ibm-cos-sdk`. For typical usage, you should install `ibm-cos-sdk` and import `TransferManager` from `ibm_cos_sdk.s3.transfer`, not directly from `ibm_cos_sdk_s3transfer`.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Ensure your Python environment is 3.8 or newer. Upgrade Python if necessary.","message":"This library requires Python 3.8 or newer. Using older Python versions will result in installation or runtime errors.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Refer to IBM COS documentation for correct endpoint URLs and ensure all required `ibm_` prefixed parameters are passed to `ibm_cos_sdk.client('s3', ...)`.","message":"IBM Cloud Object Storage (COS) requires specific configuration parameters that differ from standard AWS S3. Notably, you must provide `ibm_api_key_id`, `ibm_service_instance_id`, and a COS-specific `endpoint_url` when initializing the client.","severity":"gotcha","affected_versions":"All versions"},{"fix":"For fine-grained control, consider initializing `TransferManager` with custom `transfer_config` options. For example, `TransferConfig(max_concurrency=N)` to limit concurrent parts or `TransferConfig(multipart_threshold=M_bytes)` to adjust the multipart upload threshold.","message":"When transferring very large files or managing many concurrent transfers, `TransferManager` can consume significant system resources (memory, CPU). Monitor resource usage and adjust `max_concurrency` or `multipart_threshold` parameters if encountering issues.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-04-12T00:00:00.000Z","next_check":"2026-07-11T00:00:00.000Z"}