{"id":760,"library":"google-cloud-bigquery-datatransfer","title":"Google Cloud BigQuery Data Transfer","description":"The Google Cloud BigQuery Data Transfer API client library, currently at version 3.21.0, allows users to programmatically manage scheduled data transfers from various partner SaaS applications and other Google services into Google BigQuery. It enables automation of ETL processes and data replication. The `google-cloud-python` repository, which contains this library, generally follows a regular release cadence with frequent updates across its client libraries.","status":"active","version":"3.21.0","language":"python","source_language":"en","source_url":"https://github.com/googleapis/google-cloud-python/tree/main/packages/google-cloud-bigquery-datatransfer","tags":["google cloud","bigquery","data transfer","etl","data integration"],"install":[{"cmd":"pip install google-cloud-bigquery-datatransfer","lang":"bash","label":"Install latest version"}],"dependencies":[{"reason":"Core Google API client functionality.","package":"google-api-core"},{"reason":"Handles Google Cloud authentication.","package":"google-auth"},{"reason":"gRPC client for API communication.","package":"grpcio"},{"reason":"Provides Pythonic wrappers for Protocol Buffer messages.","package":"proto-plus"},{"reason":"Protocol Buffers for data serialization.","package":"protobuf"}],"imports":[{"note":"Client classes are typically found within a versioned submodule, like `bigquery_datatransfer_v1`.","wrong":"from google.cloud.bigquery_datatransfer import DataTransferServiceClient","symbol":"DataTransferServiceClient","correct":"from google.cloud import bigquery_datatransfer_v1\nclient = bigquery_datatransfer_v1.DataTransferServiceClient()"}],"quickstart":{"code":"import os\nfrom google.cloud import bigquery_datatransfer_v1\n\n# Set your Google Cloud Project ID and Location\n# For local development, set the GOOGLE_APPLICATION_CREDENTIALS environment variable.\n# Example: export GOOGLE_APPLICATION_CREDENTIALS=\"/path/to/your/key.json\"\n\nproject_id = os.environ.get(\"GOOGLE_CLOUD_PROJECT\", \"your-project-id\")\nlocation = \"us\"\n\nclient = bigquery_datatransfer_v1.DataTransferServiceClient()\n\nparent = client.location_path(project_id, location)\n\ntry:\n    print(f\"Listing data sources in project '{project_id}' and location '{location}':\")\n    for data_source in client.list_data_sources(parent=parent):\n        print(f\"  Name: {data_source.display_name} (ID: {data_source.data_source_id})\")\nexcept Exception as e:\n    print(f\"Error listing data sources: {e}\nEnsure the BigQuery Data Transfer API is enabled for project '{project_id}' and your credentials have sufficient permissions.\")\n","lang":"python","description":"This quickstart initializes the BigQuery Data Transfer client and lists available data sources within a specified project and location. Replace 'your-project-id' with your actual Google Cloud Project ID, or ensure the GOOGLE_CLOUD_PROJECT environment variable is set. Authentication is handled implicitly via Application Default Credentials (e.g., `gcloud auth application-default login` for local development or service account credentials on Google Cloud infrastructure). Ensure the BigQuery Data Transfer API is enabled in your Google Cloud project."},"warnings":[{"fix":"Upgrade your Python environment to Python 3.9 or newer to ensure continued compatibility and support. Check the official `google-cloud-python` documentation for the latest supported Python versions.","message":"Future versions of Google Cloud client libraries, including `google-cloud-bigquery-datatransfer`, will drop support for Python 3.7 and 3.8. Python 3.7 reached end-of-life (EOL) in June 2023, and 3.8 in October 2024. New major versions of the library will be incompatible with these EOL Python runtimes.","severity":"breaking","affected_versions":"All versions after Python 3.8 EOL (October 2024), specific breaking changes will be in new major versions."},{"fix":"Regularly monitor the BigQuery Data Transfer Service data source change log for announcements related to your specific data sources. Be prepared to update your BigQuery schemas, ETL processes, and queries as needed to accommodate schema changes.","message":"BigQuery Data Transfer Service's data source connectors (e.g., Google Ads, Display & Video 360) periodically undergo API version upgrades. These upgrades can introduce schema changes, such as new, modified, or deprecated columns, which may affect existing transfer configurations and downstream data processing pipelines.","severity":"gotcha","affected_versions":"All versions, as this is a service-level change for data connectors."},{"fix":"Ensure the service account or user initiating the transfer has the 'BigQuery Admin' (roles/bigquery.admin) role, the 'BigQuery Data Transfer Service Agent' role (roles/bigquery.datatransfer.serviceAgent), and read/write access to the source/destination datasets/tables. For external data sources, manual authorization via OAuth may also be required initially.","message":"Authentication requires appropriate IAM roles. For service accounts creating transfers, 'BigQuery Admin' and 'BigQuery Data Transfer Service Agent' roles are often necessary, in addition to permissions to access the source data. Incorrect permissions are a common cause of transfer failures.","severity":"gotcha","affected_versions":"All versions."},{"fix":"Review your data transfer configurations. To minimize costs, consider co-locating Cloud Storage buckets with your BigQuery datasets in the same region where possible.","message":"Effective February 1, 2026, BigQuery requests that read data from multi-region Cloud Storage buckets will incur Cloud Storage multi-region data transfer fees. This can significantly impact billing for transfers involving data stored across different regions.","severity":"gotcha","affected_versions":"All versions, effective Feb 1, 2026."},{"fix":"Migrate any existing transfer configurations or queries that rely on Legacy SQL to Standard SQL. Review BigQuery documentation on migrating from Legacy SQL.","message":"BigQuery will limit the use of Legacy SQL starting June 1, 2026. If a project has not used Legacy SQL between November 1, 2025, and June 1, 2026, it will no longer be able to use it. Existing workloads might continue, but new ones may fail.","severity":"deprecated","affected_versions":"All versions, impacting usage after June 1, 2026."},{"fix":"Review the Python script at the indicated line number (line 20 in this case) and correct the syntax. Ensure all strings are properly closed, parentheses/brackets are matched, and statements follow Python's grammar rules. This is a user-code error.","message":"The executed Python script contains a fundamental syntax error (e.g., an unclosed string literal, malformed statement), preventing the script from being parsed and executed correctly.","severity":"breaking","affected_versions":"All Python versions"},{"fix":"Review the Python script at the indicated line number (line 20 in this case) and ensure all f-strings and other string literals are correctly opened and closed with matching quotes. For example, 'print(f\"Error: {e}\")' instead of 'print(f\"Error: {e}')'.","message":"The script contains a SyntaxError, specifically an 'unterminated f-string literal'. This indicates a coding error in the Python script itself, where an f-string (formatted string literal) was opened but not properly closed with a matching quote.","severity":"breaking","affected_versions":"All versions (this is a coding error, not a library or environment issue)."}],"env_vars":null,"last_verified":"2026-05-12T18:41:35.324Z","next_check":"2026-06-27T00:00:00.000Z","problems":[{"fix":"Grant the appropriate IAM roles (e.g., `roles/bigquery.admin`, `roles/bigquery.dataEditor`, or specific Data Transfer Service Agent roles) to the service account or user on the relevant Google Cloud resources.","cause":"The service account or user initiating the data transfer lacks the necessary IAM permissions for the BigQuery project, dataset, or the source data.","error":"The caller does not have permission."},{"fix":"Update your code to use the `transfer_config_path` method, or refer to the library's migration guide for other API changes and use the correct path helper methods for your client library version.","cause":"This error occurs when using an outdated method from an older version of the `google-cloud-bigquery-datatransfer` client library where utility methods like `project_transfer_config_path` were removed or renamed in version 2.0.0 and above.","error":"AttributeError: 'DataTransferServiceClient' object has no attribute 'project_transfer_config_path'"},{"fix":"Enable the BigQuery Data Transfer API in the Google Cloud Console by navigating to 'APIs & Services' -> 'Enabled APIs & services' and searching for 'BigQuery Data Transfer API'.","cause":"The BigQuery Data Transfer API is not enabled for the specified Google Cloud project.","error":"BigQuery Data Transfer Service has not been used in project <project_id> before or it is disabled."},{"fix":"Ensure Application Default Credentials are set up correctly by running `gcloud auth application-default login` for local development, or by verifying that the service account used by the application has the `roles/iam.serviceAccountTokenCreator` role if impersonation is involved.","cause":"The client library could not find valid authentication credentials, or the service account used by the application lacks the necessary permissions to authenticate or create tokens for the transfer configuration.","error":"Failed to find a valid credential. The request to create a transfer config is supposed to contain an authorization code."},{"fix":"Use a supported region for the destination BigQuery dataset and the transfer configuration. Consult the official BigQuery Data Transfer documentation for a list of currently supported locations.","cause":"The specified geographic location (region or multi-region) for the BigQuery dataset or the transfer configuration is not yet supported by the BigQuery Data Transfer Service.","error":"BigQuery Data Transfer Service does not yet support location: <location>"}],"ecosystem":"pypi","meta_description":null,"install_score":100,"install_tag":"verified","quickstart_score":0,"quickstart_tag":"stale","pypi_latest":"3.22.0","cli_name":"","cli_version":null,"install_checks":{"last_tested":"2026-05-12","tag":"verified","tag_description":"installs cleanly on critical runtimes, fast import, recently tested","installed_version":null,"pypi_latest":"3.22.0","is_stale":null,"results":[{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"google-cloud-bigquery-datatransfer","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":1.67,"mem_mb":23.8,"disk_size":"68.6M"},{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"google-cloud-bigquery-datatransfer","exit_code":0,"wheel_type":null,"failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":1.63,"mem_mb":23.9,"disk_size":"67.5M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"google-cloud-bigquery-datatransfer","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":null,"install_time_s":6.2,"import_time_s":1,"mem_mb":20.5,"disk_size":"66M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"google-cloud-bigquery-datatransfer","exit_code":0,"wheel_type":null,"failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":1,"mem_mb":20.7,"disk_size":"65M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"google-cloud-bigquery-datatransfer","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":2.26,"mem_mb":25.7,"disk_size":"73.2M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"google-cloud-bigquery-datatransfer","exit_code":0,"wheel_type":null,"failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":2.58,"mem_mb":26,"disk_size":"72.1M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"google-cloud-bigquery-datatransfer","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":null,"install_time_s":5.2,"import_time_s":1.49,"mem_mb":22.6,"disk_size":"71M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"google-cloud-bigquery-datatransfer","exit_code":0,"wheel_type":null,"failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":2,"mem_mb":22.9,"disk_size":"70M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"google-cloud-bigquery-datatransfer","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":2.24,"mem_mb":24.9,"disk_size":"64.7M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"google-cloud-bigquery-datatransfer","exit_code":0,"wheel_type":null,"failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":2.51,"mem_mb":25.7,"disk_size":"63.5M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"google-cloud-bigquery-datatransfer","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":null,"install_time_s":4.4,"import_time_s":1.81,"mem_mb":21.9,"disk_size":"62M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"google-cloud-bigquery-datatransfer","exit_code":0,"wheel_type":null,"failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":2.01,"mem_mb":22.7,"disk_size":"61M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"google-cloud-bigquery-datatransfer","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":2.1,"mem_mb":25.9,"disk_size":"64.4M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"google-cloud-bigquery-datatransfer","exit_code":0,"wheel_type":null,"failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":2.61,"mem_mb":26.4,"disk_size":"63.2M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"google-cloud-bigquery-datatransfer","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":null,"install_time_s":4.8,"import_time_s":1.67,"mem_mb":22.8,"disk_size":"62M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"google-cloud-bigquery-datatransfer","exit_code":0,"wheel_type":null,"failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":2.11,"mem_mb":23.4,"disk_size":"61M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"google-cloud-bigquery-datatransfer","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":1.55,"mem_mb":23.7,"disk_size":"68.7M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"google-cloud-bigquery-datatransfer","exit_code":0,"wheel_type":null,"failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":1.56,"mem_mb":23.5,"disk_size":"67.6M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"google-cloud-bigquery-datatransfer","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":null,"install_time_s":7,"import_time_s":1.21,"mem_mb":20.4,"disk_size":"66M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"google-cloud-bigquery-datatransfer","exit_code":0,"wheel_type":null,"failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":1.06,"mem_mb":20.3,"disk_size":"65M"}]},"quickstart_checks":{"last_tested":"2026-04-24","tag":"stale","tag_description":"widespread failures or data too old to trust","results":[{"runtime":"python:3.10-alpine","exit_code":1},{"runtime":"python:3.10-slim","exit_code":1},{"runtime":"python:3.11-alpine","exit_code":1},{"runtime":"python:3.11-slim","exit_code":1},{"runtime":"python:3.12-alpine","exit_code":1},{"runtime":"python:3.12-slim","exit_code":1},{"runtime":"python:3.13-alpine","exit_code":1},{"runtime":"python:3.13-slim","exit_code":1},{"runtime":"python:3.9-alpine","exit_code":1},{"runtime":"python:3.9-slim","exit_code":1}]}}