Google Cloud Pipeline Components

2.22.0 · active · verified Sat Apr 11

This Python SDK provides a set of first-party (Google owned) pipeline components that enable users to build MLOps pipelines on Vertex AI using Kubeflow Pipelines (KFP) SDK. It integrates seamlessly with Vertex AI SDK and other Google Cloud services. The current version is 2.22.0, and releases are frequent, often aligned with KFP SDK updates.

Warnings

breaking This library is designed for Kubeflow Pipelines (KFP) v2 syntax. If migrating from KFP v1, component definition and pipeline construction syntax will be significantly different. KFP v1's `kfp.components.load_component_from_url` or direct YAML loading is generally incompatible with GCPC's object-oriented component instantiation.
Fix: Rewrite pipelines using KFP v2 DSL and directly instantiate `google-cloud-pipeline-components` classes. Consult KFP v2 and GCPC documentation for current syntax.
gotcha Proper Google Cloud IAM permissions are critical. The service account used to run the pipeline on Vertex AI (or your user credentials for local development) must have sufficient roles, such as 'Vertex AI User', 'Storage Admin' (for artifact storage), and specific permissions for any underlying GCP services your components interact with (e.g., BigQuery User, Dataflow Admin).
Fix: Ensure the service account running your Vertex AI Pipeline has all necessary IAM roles. Test permissions thoroughly in a development environment before deploying to production. Consult the Google Cloud documentation for required roles for each specific GCPC component.
gotcha Most Vertex AI services are regional. Ensure that the `project` and `location` parameters passed to GCPC components (e.g., `CustomJobOp`, `BatchPredictionJobOp`) are consistent with the region where your resources exist or where you intend to run the job. Mismatched regions can lead to resource not found errors or increased latency.
Fix: Always explicitly specify the `location` parameter for regional GCPC components and verify it matches your target Google Cloud region.

Install

pip install google-cloud-pipeline-components Install latest version

Imports

aiplatform
```
from google_cloud_pipeline_components import aiplatform
```
Commonly used for Vertex AI specific components like training jobs, batch predictions.
CustomJobOp
```
from google_cloud_pipeline_components.v1.custom_job import CustomJobOp
```
Imports a specific component. Most components are under `v1` submodules like `custom_job`, `bigquery`, `dataflow`, etc.

BigQueryQueryJobOp

from google_cloud_pipeline_components.v1.bigquery import BigQueryQueryJobOp

Example of importing a BigQuery component.

Quickstart

This quickstart defines a simple KFP pipeline using a `CustomJobOp` from `google-cloud-pipeline-components`. This component allows you to run arbitrary containerized code on Vertex AI. Remember to replace placeholder `GCP_PROJECT_ID` and `GCP_REGION` with your actual Google Cloud project and desired region, ideally using environment variables or a configuration.

import kfp
from kfp import dsl
import os

# Import a Google Cloud Pipeline Component (GCPC)
from google_cloud_pipeline_components.v1.custom_job import CustomJobOp

@dsl.pipeline(name='gcp-quickstart-pipeline', description='A simple pipeline using GCPC')
def my_gcp_pipeline():
    # Define a CustomJobOp from google-cloud-pipeline-components
    # This component launches a custom container on Vertex AI
    custom_task = CustomJobOp(
        project=os.environ.get('GCP_PROJECT_ID', 'your-gcp-project-id'),
        location=os.environ.get('GCP_REGION', 'us-central1'),
        display_name='my-first-gcp-custom-job',
        container_uri='gcr.io/cloud-aiplatform/training/tf-gpu.2-8:latest',
        command=['sh', '-c', 'echo "Hello from GCPC!"'],
        replica_count=1,
        machine_type='n1-standard-4'
    )

# To run this pipeline, you would typically compile it:
# kfp.compiler.Compiler().compile(my_gcp_pipeline, 'pipeline.json')
# And then upload and run it on Vertex AI Pipelines.

view raw JSON →