Google Cloud Orchestration Airflow API Client Library

1.20.0 · active · verified Sat Mar 28

Cloud Composer is a fully managed Apache Airflow service that simplifies the creation, scheduling, monitoring, and management of workflows. This client library provides programmatic access to the Google Cloud Orchestration Airflow API, allowing Python developers to interact with Cloud Composer environments and their underlying Airflow instances. The current version is 1.20.0, and Google Cloud client libraries typically follow a continuous release cadence with frequent updates for new features, bug fixes, and API changes.

Warnings

breaking When upgrading Cloud Composer or Airflow versions, ensure your existing DAGs and any custom PyPI packages are compatible. Incompatible changes can cause DAGs to fail. It's crucial to check the changelogs for the new Airflow version and associated provider packages.
Fix: Review the Cloud Composer and Airflow release notes for breaking changes before upgrading. Test DAGs thoroughly in a staging environment.
gotcha The library's logging events are not handled by default and may contain sensitive information. Users must explicitly configure Python's standard logging functionality to capture and handle these events.
Fix: Configure logging handlers and formatters for Python's `logging` module. Consider setting the `GOOGLE_SDK_PYTHON_LOGGING_SCOPE` environment variable for basic structured logging. Restrict access to logs containing sensitive data.
breaking The `apache-airflow-providers-google` package, which is heavily used by Cloud Composer users, has undergone several breaking changes, including the removal or renaming of various operators and hooks (e.g., `LifeSciencesRunPipelineOperator`, `BigQueryCreateEmptyTableOperator`).
Fix: Consult the changelog for `apache-airflow-providers-google` to identify replacements for deprecated operators and hooks. Migrate to the recommended alternatives, such as `Google Cloud Batch operators` or specific Vertex AI operators.
deprecated Python versions 3.6 and below are no longer supported. Using an unsupported Python version can lead to installation failures, runtime errors, and security vulnerabilities.
Fix: Upgrade your Python environment to Python 3.9 or higher. Google Cloud client libraries are compatible with all active and maintenance versions of Python.

Install

pip install google-cloud-orchestration-airflow Install latest version

Imports

EnvironmentsClient

from google.cloud.orchestration.airflow.service.v1 import EnvironmentsClient

types
```
from google.cloud.orchestration.airflow.service.v1 import types
```
Used for request/response message types (e.g., types.ListEnvironmentsRequest).

Quickstart

This quickstart code initializes the `EnvironmentsClient` and attempts to list all Cloud Composer environments within a specified Google Cloud project and location. Authentication is handled implicitly by the Google Cloud client libraries, typically through environment variables (e.g., `GOOGLE_APPLICATION_CREDENTIALS`) or the default service account if running on GCP. Users should replace `your-gcp-project-id` and specify a `location` (e.g., 'us-central1', 'europe-west1') where their Composer environments might reside.

import os
from google.cloud.orchestration.airflow.service.v1 import EnvironmentsClient
from google.cloud.orchestration.airflow.service.v1 import types

def list_composer_environments(project_id: str, location: str):
    """Lists Cloud Composer environments in a given project and location."""
    client = EnvironmentsClient()
    parent = f"projects/{project_id}/locations/{location}"

    try:
        request = types.ListEnvironmentsRequest(parent=parent)
        page_result = client.list_environments(request=request)

        print(f"Cloud Composer Environments in {location}:")
        found_any = False
        for environment in page_result:
            print(f"  - {environment.name} (State: {environment.state.name})")
            found_any = True
        if not found_any:
            print("  No environments found.")

    except Exception as e:
        print(f"Error listing environments: {e}")

if __name__ == "__main__":
    project_id = os.environ.get('GOOGLE_CLOUD_PROJECT', 'your-gcp-project-id')
    location = os.environ.get('GOOGLE_CLOUD_LOCATION', 'us-central1') # e.g., 'us-central1', 'europe-west1'

    if project_id == 'your-gcp-project-id':
        print("Please set the GOOGLE_CLOUD_PROJECT environment variable or replace 'your-gcp-project-id'.")
    elif location == 'us-central1':
        print("Please set the GOOGLE_CLOUD_LOCATION environment variable or choose your desired location.")
    else:
        list_composer_environments(project_id, location)

view raw JSON →