{"id":988,"library":"google-cloud-storage-transfer","title":"Google Cloud Storage Transfer API Client Library","description":"The `google-cloud-storage-transfer` library is the official Python client for the Google Cloud Storage Transfer Service. It enables programmatic control over data transfers to and from Google Cloud Storage, supporting various sources including other cloud providers and on-premises systems. Currently at version 1.20.0, this library adheres to Google Cloud's frequent release cadence, often receiving updates alongside other Python client libraries in the `googleapis/google-cloud-python` monorepo.","status":"active","version":"1.20.0","language":"python","source_language":"en","source_url":"https://github.com/googleapis/google-cloud-python/tree/main/packages/google-cloud-storage-transfer","tags":["google cloud","storage transfer","data transfer","gcs"],"install":[{"cmd":"pip install google-cloud-storage-transfer","lang":"bash","label":"Install stable version"}],"dependencies":[],"imports":[{"note":"The `_v1` suffix is often for older, more specific API versions; the top-level import is preferred for the latest stable client.","wrong":"from google.cloud.storagetransfer_v1 import StorageTransferServiceClient","symbol":"StorageTransferServiceClient","correct":"from google.cloud.storage_transfer import StorageTransferServiceClient"}],"quickstart":{"code":"import os\nfrom google.cloud.storage_transfer import StorageTransferServiceClient\n\ndef create_and_run_gcs_to_gcs_transfer_job(\n    project_id: str,\n    source_bucket_name: str,\n    sink_bucket_name: str,\n    job_description: str,\n):\n    \"\"\"Creates and runs a one-time transfer job between two GCS buckets.\"\"\"\n    client = StorageTransferServiceClient()\n\n    # Transfer job configuration\n    transfer_job = {\n        \"project_id\": project_id,\n        \"description\": job_description,\n        \"transfer_spec\": {\n            \"gcs_data_source\": {\"bucket_name\": source_bucket_name},\n            \"gcs_data_sink\": {\"bucket_name\": sink_bucket_name},\n        },\n        \"status\": \"ENABLED\",  # Job is created in an enabled state\n    }\n\n    # Create the transfer job\n    # The API might create the job as DISABLED and then ENABLE it, or directly ENABLED.\n    # For a 'run now' quickstart, we often set it to ENABLED at creation.\n    try:\n        created_job = client.create_transfer_job(transfer_job=transfer_job)\n        print(f\"Created transfer job: {created_job.name}\")\n\n        # If the job is not already IN_PROGRESS (e.g., if set to ENABLED but not yet started)\n        # we can explicitly run it. For a one-time job set to ENABLED, it should start automatically.\n        # However, for demonstration, an explicit run call can be shown for clarity if desired\n        # for existing jobs, but 'create' with ENABLED often triggers it immediately for one-time.\n        print(f\"Transfer job '{created_job.name}' initiated.\")\n        # For more complex job management (e.g., recurrent jobs), you'd interact more with job status and run_transfer_job\n\n    except Exception as e:\n        print(f\"Error creating or running transfer job: {e}\")\n\n# Example usage (replace with your actual project and bucket names)\nif __name__ == \"__main__\":\n    # Ensure these environment variables are set or replace with actual values\n    # For local development, `gcloud auth application-default login` often provides credentials.\n    PROJECT_ID = os.environ.get(\"GCP_PROJECT_ID\", \"your-gcp-project-id\")\n    SOURCE_BUCKET = os.environ.get(\"GCP_SOURCE_BUCKET\", \"your-source-gcs-bucket\")\n    SINK_BUCKET = os.environ.get(\"GCP_SINK_BUCKET\", \"your-sink-gcs-bucket\")\n    JOB_DESCRIPTION = \"My Python quickstart GCS to GCS transfer\"\n\n    if PROJECT_ID == \"your-gcp-project-id\" or SOURCE_BUCKET == \"your-source-gcs-bucket\" or SINK_BUCKET == \"your-sink-gcs-bucket\":\n        print(\"Please set GCP_PROJECT_ID, GCP_SOURCE_BUCKET, and GCP_SINK_BUCKET environment variables or replace placeholder values.\")\n    else:\n        create_and_run_gcs_to_gcs_transfer_job(\n            PROJECT_ID, SOURCE_BUCKET, SINK_BUCKET, JOB_DESCRIPTION\n        )\n","lang":"python","description":"This quickstart demonstrates how to create and immediately run a one-time transfer job to move data from a Google Cloud Storage source bucket to a Google Cloud Storage sink bucket. It requires enabling the Storage Transfer Service API, setting up Application Default Credentials, and ensuring the service account has appropriate permissions to both buckets. Replace placeholder variables with your Google Cloud Project ID and GCS bucket names."},"warnings":[{"fix":"Go to the Google Cloud Console, navigate to 'APIs & Services' > 'Library', search for 'Storage Transfer API', and enable it. For authentication, run `gcloud auth application-default login` locally, or ensure your deployment environment (e.g., GCE, Cloud Run) has a service account with necessary roles.","message":"The Storage Transfer Service API must be explicitly enabled in your Google Cloud Project before use. Additionally, ensure proper authentication is configured, ideally using Application Default Credentials (ADC).","severity":"gotcha","affected_versions":"All versions"},{"fix":"Review the official 'Quotas & Limits' documentation for Storage Transfer Service. Implement exponential backoff for retries and design transfer jobs to stay within known limits. For very large transfers, consider splitting the data into multiple jobs.","message":"The service has specific quotas and limits, including rate limits (e.g., 600 requests/min/project) and a 5 TiB maximum object size for transfers to Cloud Storage. Exceeding these limits can lead to failures or throttling.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Grant the necessary IAM roles to the principal performing the transfer. For example, 'Storage Transfer Admin' for managing jobs, and 'Storage Object Viewer'/'Storage Object Creator'/'Storage Object Admin' for source/sink buckets as appropriate.","message":"Insufficient IAM permissions are a common cause of transfer failures. The service account or user initiating the transfer needs permissions to create and manage transfer jobs, and read/write access to the source and destination resources.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Update your dependencies to `google-cloud-storage-transfer` and refactor client instantiation and method calls according to the Cloud Client Library's patterns. Consult the 'Migrating to the Storage Transfer Service Cloud Client Library' guide for specific changes.","message":"If you are migrating from or encounter older code using the `google-api-services-storagetransfer` library, be aware that it's a legacy Google API Client Library. The `google-cloud-storage-transfer` is the recommended Cloud Client Library.","severity":"breaking","affected_versions":"Projects using legacy libraries"},{"fix":"Configure your transfer jobs to store logs (e.g., using `--log-dir` in gcloud CLI or equivalent API parameter). Review these logs in Cloud Logging for detailed error messages, such as permission issues, missing files, or network problems.","message":"When troubleshooting failed transfer jobs, it's crucial to enable and inspect logs to understand the root cause, especially for agent-based transfers (on-premises to cloud).","severity":"gotcha","affected_versions":"All versions"},{"fix":"Upgrade your Python environment to version 3.10 or newer. After upgrading Python, update all Google Cloud Client Libraries and their core dependencies (e.g., `google-api-core`, `google-auth`) to their latest compatible versions using `pip install --upgrade google-cloud-storage-transfer google-api-core google-auth`.","message":"Using Python 3.9 or older versions with `google-cloud-storage-transfer` (and its dependencies like `google-api-core`, `google-auth`) will trigger `FutureWarning`s indicating that these Python versions are unsupported and will receive limited or no further updates. This may lead to unexpected behavior or lack of critical fixes in the future.","severity":"gotcha","affected_versions":"Projects using Python 3.9 or older"},{"fix":"Ensure that the Google Cloud Project ID, source bucket name, and destination bucket name are correctly provided to your application, typically via environment variables (e.g., `GCP_PROJECT_ID`, `GCP_SOURCE_BUCKET`, `GCP_SINK_BUCKET`), or passed explicitly as parameters to the Storage Transfer Service client methods.","message":"Users must provide valid Google Cloud Project ID, source bucket name, and destination bucket name for transfer operations. These are fundamental identifiers required to configure and execute transfer jobs.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-05-12T22:16:15.326Z","next_check":"2026-06-27T00:00:00.000Z","problems":[{"fix":"pip install google-cloud-storage-transfer","cause":"The `google-cloud-storage-transfer` Python package is not installed or not accessible in the current Python environment.","error":"ModuleNotFoundError: No module named 'google.cloud.storage_transfer'"},{"fix":"Grant the `Storage Transfer Service Agent` role (`roles/storagetransfer.serviceAgent`) to the service account on the project, and the `Storage Admin` role (`roles/storage.admin`) on all relevant source and destination buckets.","cause":"The Google-managed service account used by Storage Transfer Service (typically `project-PROJECT_NUMBER@storage-transfer-service.iam.gserviceaccount.com`) does not have the necessary IAM permissions to access the source or destination resources.","error":"Service lacked sufficient permissions, SERVICE_PERMISSION_FAILURE"},{"fix":"Verify the spelling and existence of the destination bucket, and ensure the Storage Transfer Service agent has the `Storage Admin` role on the destination bucket.","cause":"The specified destination bucket in the transfer job configuration either does not exist, or the Storage Transfer Service agent does not have permission to view or write to it.","error":"The destination bucket doesn't exist in Cloud Storage."}],"ecosystem":"pypi","meta_description":null,"install_score":100,"install_tag":"verified","quickstart_score":null,"quickstart_tag":null,"pypi_latest":"1.20.0","cli_name":"","install_checks":{"last_tested":"2026-05-12","tag":"verified","tag_description":"installs cleanly on critical runtimes, fast import, recently tested","results":[{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":null,"import_time_s":1.87,"mem_mb":24.9,"disk_size":"68.6M"},{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":1.72,"mem_mb":24.5,"disk_size":"67.4M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":5.8,"import_time_s":1.11,"mem_mb":21.1,"disk_size":"66M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":1.07,"mem_mb":20.8,"disk_size":"65M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":null,"import_time_s":2.51,"mem_mb":26.8,"disk_size":"73.1M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":2.63,"mem_mb":26.5,"disk_size":"72.0M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":5.1,"import_time_s":1.66,"mem_mb":23.3,"disk_size":"71M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":1.52,"mem_mb":23,"disk_size":"70M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":null,"import_time_s":2.57,"mem_mb":26.6,"disk_size":"64.6M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":2.69,"mem_mb":26.4,"disk_size":"63.5M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":4.2,"import_time_s":1.95,"mem_mb":22.6,"disk_size":"62M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":2.11,"mem_mb":22.3,"disk_size":"61M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":null,"import_time_s":2.37,"mem_mb":27,"disk_size":"64.3M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":2.54,"mem_mb":26.7,"disk_size":"63.1M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":4.4,"import_time_s":1.88,"mem_mb":23.5,"disk_size":"62M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":2.04,"mem_mb":23.2,"disk_size":"61M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":null,"import_time_s":1.73,"mem_mb":24.8,"disk_size":"68.6M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":1.62,"mem_mb":24.6,"disk_size":"67.5M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":6.7,"import_time_s":1.43,"mem_mb":21.1,"disk_size":"66M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":1.21,"mem_mb":20.9,"disk_size":"65M"}]},"quickstart_checks":{"last_tested":"2026-04-24","tag":null,"tag_description":null,"results":[{"runtime":"python:3.10-alpine","exit_code":0},{"runtime":"python:3.10-slim","exit_code":0},{"runtime":"python:3.11-alpine","exit_code":0},{"runtime":"python:3.11-slim","exit_code":0},{"runtime":"python:3.12-alpine","exit_code":0},{"runtime":"python:3.12-slim","exit_code":0},{"runtime":"python:3.13-alpine","exit_code":0},{"runtime":"python:3.13-slim","exit_code":0},{"runtime":"python:3.9-alpine","exit_code":0},{"runtime":"python:3.9-slim","exit_code":0}]}}