Kubeflow Pipelines Pipeline Specification

2.16.0 · active · verified Thu Apr 09

kfp-pipeline-spec defines the canonical data structures and protobuf messages for Kubeflow Pipelines (KFP) v2 pipeline specifications. It is a core component of the KFP SDK, providing the underlying schema for pipeline definitions compiled by `kfp` and executed on the KFP backend. The library is actively maintained, with releases frequently synchronised with the main KFP SDK, currently at version 2.16.0.

Warnings

breaking The pipeline specification format underwent significant changes between Kubeflow Pipelines v1 and v2. `kfp-pipeline-spec` is designed for KFP v2 and is not compatible with pipeline definitions or compilers from KFP v1.
Fix: Ensure your KFP SDK and backend are both on version 2.x or higher when using `kfp-pipeline-spec`. Old v1 pipelines need to be migrated or re-authored using KFP v2 SDK.
gotcha `kfp-pipeline-spec` is primarily a definitions library for the KFP v2 pipeline schema. Most pipeline authors will interact with this schema indirectly through the `kfp` SDK's `kfp.dsl` and `kfp.compiler` modules, rather than directly constructing protobuf objects from `kfp_pipeline_spec`.
Fix: For authoring pipelines, use the high-level `kfp` SDK (e.g., `@dsl.pipeline`, `@dsl.component`). Directly manipulating `kfp_pipeline_spec` is typically reserved for advanced use cases like custom compilers, introspection, or validation tools.
gotcha The KFP ecosystem consists of several related Python packages (`kfp`, `kfp-pipeline-spec`, `kfp-server-api`, `kfp-kubernetes`). It's a common mistake to confuse which package provides specific classes or functionalities. For example, the `Compiler` is in `kfp`, while `PipelineSpec` definition is in `kfp-pipeline-spec`.
Fix: Always refer to the official KFP documentation for correct import paths and module responsibilities. Generally, `kfp` is for pipeline authoring, `kfp-pipeline-spec` for schema definitions, and `kfp-server-api` for interacting with the KFP API server.

Install

pip install kfp-pipeline-spec==2.16.0 Install specific version

Imports

PipelineSpec
```
from kfp_pipeline_spec.pipeline_spec_pb2 import PipelineSpec
```
The v2 compiler path was common in older pre-GA KFP v2 releases; for direct access to the spec protobufs, import from `kfp_pipeline_spec`.

ContainerSpec

from kfp_pipeline_spec.pipeline_spec_pb2 import ContainerSpec

Quickstart

This quickstart demonstrates how to programmatically construct a barebones `PipelineSpec` object using the protobuf definitions from `kfp-pipeline-spec`. While most users generate this spec via the `kfp` SDK's compiler, this shows direct interaction with the underlying data model. It then serializes the spec to JSON, which is a common output format for compiled KFP pipelines.

import kfp_pipeline_spec.pipeline_spec_pb2 as pb
from google.protobuf import json_format

# Create a minimal KFP PipelineSpec object directly using the protobuf definitions
pipeline_spec = pb.PipelineSpec(
    pipeline_info=pb.PipelineInfo(name="my-direct-pipeline"),
    root=pb.PipelineTaskSpec(
        executor_label="my-executor",
        inputs=pb.TaskInputsSpec(),
        outputs=pb.TaskOutputsSpec(),
    ),
    components={
        "my-executor": pb.ComponentSpec(
            executor_label="my-executor",
            input_definitions=pb.ComponentInputsSpec(),
            output_definitions=pb.ComponentOutputsSpec(),
            implementation=pb.ComponentSpec.ContainerImplementation(
                container=pb.ContainerSpec(
                    image="python:3.9-slim",
                    command=["python", "-c", "print('Hello from kfp-pipeline-spec!')"]
                )
            )
        )
    }
)

# Serialize the PipelineSpec to JSON (as it would be in a compiled YAML file)
json_output = json_format.MessageToJson(pipeline_spec, indent=2)
print(json_output)

# To verify it's a valid spec object
assert isinstance(pipeline_spec, pb.PipelineSpec)
print(f"Pipeline name: {pipeline_spec.pipeline_info.name}")

view raw JSON →