{"id":493,"library":"google-cloud-dataform","title":"Google Cloud Dataform","description":"Google Cloud Dataform is a service for developing, creating, documenting, testing, and updating curated tables in BigQuery. This Python client library (version 0.9.0) provides an interface to interact with the Dataform API. Google Cloud client libraries typically follow a regular release cadence, often coinciding with API updates or bug fixes, and are generally well-maintained.","status":"active","version":"0.9.0","language":"python","source_language":"en","source_url":"https://github.com/googleapis/google-cloud-python/tree/main/packages/google-cloud-dataform","tags":["google cloud","dataform","etl","data orchestration","bigquery"],"install":[{"cmd":"pip install google-cloud-dataform","lang":"bash","label":"Install latest version"}],"dependencies":[{"reason":"Core library for Google Cloud API clients.","package":"google-api-core","optional":false},{"reason":"Handles authentication with Google Cloud services.","package":"google-auth","optional":false},{"reason":"Provides Pythonic wrappers around Protocol Buffer messages.","package":"proto-plus","optional":false}],"imports":[{"note":"Google Cloud client libraries typically organize clients under versioned submodules (e.g., `_v1beta`). Direct import from `google.cloud.dataform` is incorrect.","wrong":"from google.cloud.dataform import DataformClient","symbol":"DataformClient","correct":"from google.cloud import dataform_v1beta\nclient = dataform_v1beta.DataformClient()"}],"quickstart":{"code":"import os\nfrom google.cloud import dataform_v1beta\n\n# Set your Google Cloud Project ID and Location\n# For example, 'my-project-id' and 'us-central1'\nproject_id = os.environ.get('GOOGLE_CLOUD_PROJECT', 'your-project-id')\nlocation = 'us-central1' # Replace with your Dataform repository location\n\ndef list_dataform_repositories(project_id: str, location: str):\n    \"\"\"Lists Dataform repositories in a given project and location.\"\"\"\n    client = dataform_v1beta.DataformClient()\n    parent = f\"projects/{project_id}/locations/{location}\"\n\n    print(f\"Listing Dataform repositories in {parent}:\")\n    try:\n        for repository in client.list_repositories(parent=parent):\n            print(f\"- Repository: {repository.name}\")\n    except Exception as e:\n        print(f\"Error listing repositories: {e}\")\n        print(\"Please ensure the Dataform API is enabled and your account has the necessary permissions.\")\n\nif __name__ == \"__main__\":\n    list_dataform_repositories(project_id, location)","lang":"python","description":"This quickstart initializes a Dataform client and lists all Dataform repositories within a specified Google Cloud project and location. Ensure your `GOOGLE_CLOUD_PROJECT` environment variable is set or replace 'your-project-id' with your actual project ID, and set the correct location. Authentication is handled automatically via Application Default Credentials (ADC)."},"warnings":[{"fix":"Migrate to using custom service accounts for Dataform repositories and ensure `roles/iam.serviceAccountUser` is granted to them. Refer to Google Cloud documentation on 'Use strict act-as mode' and 'Verify act-as permissions'.","message":"As of January 19, 2026, Dataform workflows, BigQuery notebooks, pipelines, and data preparations enforce strict act-as mode at the project level. This requires using custom service accounts instead of the default Dataform service agent across all repositories and granting the Service Account User role (`roles/iam.serviceAccountUser`) to the default Dataform service agent and relevant principals to avoid failures.","severity":"breaking","affected_versions":">= 0.9.0"},{"fix":"Migrate any existing legacy Dataform projects to Dataform in Google Cloud Platform. Refer to the official migration guide.","message":"Legacy Dataform was deprecated on February 26, 2024. Projects running on legacy Dataform cannot be accessed after this date.","severity":"deprecated","affected_versions":"N/A (affects old Dataform product, not this client library directly)"},{"fix":"Pin your dependency to specific patch versions (e.g., `google-cloud-dataform==0.9.0`) to prevent unexpected breakage during upgrades, and thoroughly test minor version updates.","message":"The `google-cloud-dataform` library is currently in `0.x.x` version series, indicating that its API surface might still undergo backward-incompatible changes between minor versions (e.g., 0.8.0 to 0.9.0). While Google aims for stability, pre-1.0 releases do not strictly adhere to semantic versioning guarantees.","severity":"gotcha","affected_versions":"< 1.0.0"},{"fix":"Do not build critical system logic solely dependent on the exact format or content of library-generated log messages. Configure standard Python logging handlers to capture logs at appropriate levels.","message":"The library's internal logging events (occurrence, level, content) may be refined without being flagged as breaking changes. Avoid depending on the immutability of these logging events.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Upgrade your Python environment to version 3.7 or higher. All current active and maintenance versions of Python (>=3.7) are compatible.","message":"Python versions 3.6 and below are no longer supported by Google Cloud client libraries, including `google-cloud-dataform`.","severity":"breaking","affected_versions":"<= 3.6"},{"fix":"Ensure the `google-cloud-dataform` library is installed using `pip install google-cloud-dataform`. If already installed, ensure it is up-to-date or compatible with the desired API version (`dataform_v1beta`). Verify installed packages with `pip freeze` and consult the library's documentation or changelog for module availability and import paths.","message":"The required `google-cloud-dataform` client library, or the specific version containing the `dataform_v1beta` submodule, may not be installed or is incompatible with the import statement. Importing client components directly from `google.cloud` (e.g., `from google.cloud import dataform_v1beta`) depends on the individual client library being correctly installed and compatible.","severity":"breaking","affected_versions":"All versions"},{"fix":"Use `from google.cloud.dataform_v1beta import DataformClient` to import the Dataform v1beta client, or `from google.cloud import dataform` for the stable client if that is desired, followed by `dataform.DataformClient`.","message":"Attempting to import `dataform_v1beta` directly from `google.cloud` (e.g., `from google.cloud import dataform_v1beta`) will result in an `ImportError`. The Dataform v1beta client is typically found within its own subpackage.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-05-12T14:18:33.742Z","next_check":"2026-06-26T00:00:00.000Z","problems":[{"fix":"Grant the Dataform service account (or your user account, if using user credentials) the `BigQuery Job User` role or the specific `bigquery.jobs.create` permission on the Google Cloud project.","cause":"The Dataform service account or the user triggering the Dataform job lacks the necessary IAM permissions to create BigQuery jobs in the specified project.","error":"Access Denied: Project PROJECT_ID : User does not have bigquery.jobs.create permission in project PROJECT_ID."},{"fix":"Verify the Git access token or SSH private key stored in Secret Manager, ensure it has the correct permissions for the repository, check for extra spaces/line breaks in the secret, and confirm no firewall rules are blocking SSH (port 22) or HTTPS traffic. If using SAML SSO, ensure the SSH key is authorized in the Git provider.","cause":"Dataform is unable to authenticate with the linked Git repository, often due to an incorrect, expired, or unauthorized SSH key or access token, or network restrictions.","error":"An error occurred during computation of the authentication status"},{"fix":"Ensure that `npm install` has been run in your Dataform project directory to install dependencies, and explicitly set a recent, compatible version of `@dataform/core` (e.g., `\"@dataform/core\": \"3.0.0\"` or later) in your `package.json` file, then reinstall packages.","cause":"The `@dataform/core` package is either missing, corrupted, or an incompatible version is installed, preventing Dataform from compiling SQLX files.","error":"Failed to find the transpiler exported from @dataform/core. Ensure packages are installed and upgrade to a recent version."},{"fix":"Reduce the number of parallel queries in Dataform by categorizing actions with tags and running only selected tags, introducing dependencies between actions, or dividing executions of actions between different Google Cloud projects.","cause":"The number of concurrent API requests that Dataform sends to BigQuery has exceeded the BigQuery API quota for your project or user, leading to throttling.","error":"Quota exceeded: Your user_method exceeded quota for concurrent api requests per user per method."},{"fix":"Review the Dataform configuration files for syntax errors, ensuring that all properties (like `type`, `tags`, etc.) have correctly formatted values and are placed in valid contexts according to Dataform's documentation.","cause":"This error typically occurs when there is a syntax or type mismatch in the configuration of a Dataform file (e.g., `sqlx`, `yaml`, or `json`), especially when a property expects a specific type but receives another.","error":"Dataform encountered an error: Unexpected property \"type\", or property value type of \"string\" is incorrect."}],"ecosystem":"pypi","meta_description":null,"install_score":0,"install_tag":"stale","quickstart_score":0,"quickstart_tag":"stale","pypi_latest":null,"install_checks":{"last_tested":"2026-05-12","tag":"stale","tag_description":"widespread failures or data too old to trust","results":[{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"default","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"default","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"default","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"default","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"default","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"default","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"default","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"default","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"default","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"default","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null}]},"quickstart_checks":{"last_tested":"2026-04-23","tag":"stale","tag_description":"widespread failures or data too old to trust","results":[{"runtime":"python:3.10-alpine","exit_code":1},{"runtime":"python:3.10-slim","exit_code":1},{"runtime":"python:3.11-alpine","exit_code":1},{"runtime":"python:3.11-slim","exit_code":1},{"runtime":"python:3.12-alpine","exit_code":1},{"runtime":"python:3.12-slim","exit_code":1},{"runtime":"python:3.13-alpine","exit_code":1},{"runtime":"python:3.13-slim","exit_code":1},{"runtime":"python:3.9-alpine","exit_code":1},{"runtime":"python:3.9-slim","exit_code":1}]}}