Google Cloud BigQuery Data Transfer

3.21.0 · active · verified Sun Mar 29

The Google Cloud BigQuery Data Transfer API client library, currently at version 3.21.0, allows users to programmatically manage scheduled data transfers from various partner SaaS applications and other Google services into Google BigQuery. It enables automation of ETL processes and data replication. The `google-cloud-python` repository, which contains this library, generally follows a regular release cadence with frequent updates across its client libraries.

Warnings

breaking Future versions of Google Cloud client libraries, including `google-cloud-bigquery-datatransfer`, will drop support for Python 3.7 and 3.8. Python 3.7 reached end-of-life (EOL) in June 2023, and 3.8 in October 2024. New major versions of the library will be incompatible with these EOL Python runtimes.
Fix: Upgrade your Python environment to Python 3.9 or newer to ensure continued compatibility and support. Check the official `google-cloud-python` documentation for the latest supported Python versions.
gotcha BigQuery Data Transfer Service's data source connectors (e.g., Google Ads, Display & Video 360) periodically undergo API version upgrades. These upgrades can introduce schema changes, such as new, modified, or deprecated columns, which may affect existing transfer configurations and downstream data processing pipelines.
Fix: Regularly monitor the BigQuery Data Transfer Service data source change log for announcements related to your specific data sources. Be prepared to update your BigQuery schemas, ETL processes, and queries as needed to accommodate schema changes.
gotcha Authentication requires appropriate IAM roles. For service accounts creating transfers, 'BigQuery Admin' and 'BigQuery Data Transfer Service Agent' roles are often necessary, in addition to permissions to access the source data. Incorrect permissions are a common cause of transfer failures.
Fix: Ensure the service account or user initiating the transfer has the 'BigQuery Admin' (roles/bigquery.admin) role, the 'BigQuery Data Transfer Service Agent' role (roles/bigquery.datatransfer.serviceAgent), and read/write access to the source/destination datasets/tables. For external data sources, manual authorization via OAuth may also be required initially.
gotcha Effective February 1, 2026, BigQuery requests that read data from multi-region Cloud Storage buckets will incur Cloud Storage multi-region data transfer fees. This can significantly impact billing for transfers involving data stored across different regions.
Fix: Review your data transfer configurations. To minimize costs, consider co-locating Cloud Storage buckets with your BigQuery datasets in the same region where possible.
deprecated BigQuery will limit the use of Legacy SQL starting June 1, 2026. If a project has not used Legacy SQL between November 1, 2025, and June 1, 2026, it will no longer be able to use it. Existing workloads might continue, but new ones may fail.
Fix: Migrate any existing transfer configurations or queries that rely on Legacy SQL to Standard SQL. Review BigQuery documentation on migrating from Legacy SQL.

Install

pip install google-cloud-bigquery-datatransfer Install latest version

Imports

DataTransferServiceClient
```
from google.cloud import bigquery_datatransfer_v1
client = bigquery_datatransfer_v1.DataTransferServiceClient()
```
Client classes are typically found within a versioned submodule, like `bigquery_datatransfer_v1`.

Quickstart

This quickstart initializes the BigQuery Data Transfer client and lists available data sources within a specified project and location. Replace 'your-project-id' with your actual Google Cloud Project ID, or ensure the GOOGLE_CLOUD_PROJECT environment variable is set. Authentication is handled implicitly via Application Default Credentials (e.g., `gcloud auth application-default login` for local development or service account credentials on Google Cloud infrastructure). Ensure the BigQuery Data Transfer API is enabled in your Google Cloud project.

import os
from google.cloud import bigquery_datatransfer_v1

# Set your Google Cloud Project ID and Location
# For local development, set the GOOGLE_APPLICATION_CREDENTIALS environment variable.
# Example: export GOOGLE_APPLICATION_CREDENTIALS="/path/to/your/key.json"

project_id = os.environ.get("GOOGLE_CLOUD_PROJECT", "your-project-id")
location = "us"

client = bigquery_datatransfer_v1.DataTransferServiceClient()

parent = client.location_path(project_id, location)

try:
    print(f"Listing data sources in project '{project_id}' and location '{location}':")
    for data_source in client.list_data_sources(parent=parent):
        print(f"  Name: {data_source.display_name} (ID: {data_source.data_source_id})")
except Exception as e:
    print(f"Error listing data sources: {e}
Ensure the BigQuery Data Transfer API is enabled for project '{project_id}' and your credentials have sufficient permissions.")

view raw JSON →