Google Cloud Storage Transfer API Client Library

Warnings

• gotcha The Storage Transfer Service API must be explicitly enabled in your Google Cloud Project before use. Additionally, ensure proper authentication is configured, ideally using Application Default Credentials (ADC).
  Fix: Go to the Google Cloud Console, navigate to 'APIs & Services' > 'Library', search for 'Storage Transfer API', and enable it. For authentication, run `gcloud auth application-default login` locally, or ensure your deployment environment (e.g., GCE, Cloud Run) has a service account with necessary roles.
• gotcha The service has specific quotas and limits, including rate limits (e.g., 600 requests/min/project) and a 5 TiB maximum object size for transfers to Cloud Storage. Exceeding these limits can lead to failures or throttling.
  Fix: Review the official 'Quotas & Limits' documentation for Storage Transfer Service. Implement exponential backoff for retries and design transfer jobs to stay within known limits. For very large transfers, consider splitting the data into multiple jobs.
• gotcha Insufficient IAM permissions are a common cause of transfer failures. The service account or user initiating the transfer needs permissions to create and manage transfer jobs, and read/write access to the source and destination resources.
  Fix: Grant the necessary IAM roles to the principal performing the transfer. For example, 'Storage Transfer Admin' for managing jobs, and 'Storage Object Viewer'/'Storage Object Creator'/'Storage Object Admin' for source/sink buckets as appropriate.
• breaking If you are migrating from or encounter older code using the `google-api-services-storagetransfer` library, be aware that it's a legacy Google API Client Library. The `google-cloud-storage-transfer` is the recommended Cloud Client Library.
  Fix: Update your dependencies to `google-cloud-storage-transfer` and refactor client instantiation and method calls according to the Cloud Client Library's patterns. Consult the 'Migrating to the Storage Transfer Service Cloud Client Library' guide for specific changes.
• gotcha When troubleshooting failed transfer jobs, it's crucial to enable and inspect logs to understand the root cause, especially for agent-based transfers (on-premises to cloud).
  Fix: Configure your transfer jobs to store logs (e.g., using `--log-dir` in gcloud CLI or equivalent API parameter). Review these logs in Cloud Logging for detailed error messages, such as permission issues, missing files, or network problems.

Install

• pip install google-cloud-storage-transfer Install stable version

Imports

• StorageTransferServiceClient

  from google.cloud.storage_transfer import StorageTransferServiceClient

  The `_v1` suffix is often for older, more specific API versions; the top-level import is preferred for the latest stable client.

Quickstart

This quickstart demonstrates how to create and immediately run a one-time transfer job to move data from a Google Cloud Storage source bucket to a Google Cloud Storage sink bucket. It requires enabling the Storage Transfer Service API, setting up Application Default Credentials, and ensuring the service account has appropriate permissions to both buckets. Replace placeholder variables with your Google Cloud Project ID and GCS bucket names.

import os
from google.cloud.storage_transfer import StorageTransferServiceClient

def create_and_run_gcs_to_gcs_transfer_job(
    project_id: str,
    source_bucket_name: str,
    sink_bucket_name: str,
    job_description: str,
):
    """Creates and runs a one-time transfer job between two GCS buckets."""
    client = StorageTransferServiceClient()

    # Transfer job configuration
    transfer_job = {
        "project_id": project_id,
        "description": job_description,
        "transfer_spec": {
            "gcs_data_source": {"bucket_name": source_bucket_name},
            "gcs_data_sink": {"bucket_name": sink_bucket_name},
        },
        "status": "ENABLED",  # Job is created in an enabled state
    }

    # Create the transfer job
    # The API might create the job as DISABLED and then ENABLE it, or directly ENABLED.
    # For a 'run now' quickstart, we often set it to ENABLED at creation.
    try:
        created_job = client.create_transfer_job(transfer_job=transfer_job)
        print(f"Created transfer job: {created_job.name}")

        # If the job is not already IN_PROGRESS (e.g., if set to ENABLED but not yet started)
        # we can explicitly run it. For a one-time job set to ENABLED, it should start automatically.
        # However, for demonstration, an explicit run call can be shown for clarity if desired
        # for existing jobs, but 'create' with ENABLED often triggers it immediately for one-time.
        print(f"Transfer job '{created_job.name}' initiated.")
        # For more complex job management (e.g., recurrent jobs), you'd interact more with job status and run_transfer_job

    except Exception as e:
        print(f"Error creating or running transfer job: {e}")

# Example usage (replace with your actual project and bucket names)
if __name__ == "__main__":
    # Ensure these environment variables are set or replace with actual values
    # For local development, `gcloud auth application-default login` often provides credentials.
    PROJECT_ID = os.environ.get("GCP_PROJECT_ID", "your-gcp-project-id")
    SOURCE_BUCKET = os.environ.get("GCP_SOURCE_BUCKET", "your-source-gcs-bucket")
    SINK_BUCKET = os.environ.get("GCP_SINK_BUCKET", "your-sink-gcs-bucket")
    JOB_DESCRIPTION = "My Python quickstart GCS to GCS transfer"

    if PROJECT_ID == "your-gcp-project-id" or SOURCE_BUCKET == "your-source-gcs-bucket" or SINK_BUCKET == "your-sink-gcs-bucket":
        print("Please set GCP_PROJECT_ID, GCP_SOURCE_BUCKET, and GCP_SINK_BUCKET environment variables or replace placeholder values.")
    else:
        create_and_run_gcs_to_gcs_transfer_job(
            PROJECT_ID, SOURCE_BUCKET, SINK_BUCKET, JOB_DESCRIPTION
        )