{"id":1294,"library":"kfp","title":"Kubeflow Pipelines SDK","description":"The Kubeflow Pipelines SDK (kfp), currently at version 2.16.0, is a Python library for building and deploying portable, scalable machine learning workflows based on Docker containers within the Kubeflow project. It allows users to compose multi-step workflows (pipelines) as a graph of containerized tasks using Python code and/or YAML. Releases are frequent, often bundling the SDK with related components like `kfp-pipeline-spec`, `kfp-server-api`, and `kfp-kubernetes`.","status":"active","version":"2.16.0","language":"en","source_language":"en","source_url":"https://github.com/kubeflow/pipelines/tree/master/sdk","tags":["machine learning","mlopps","kubernetes","pipeline","workflow"],"install":[{"cmd":"pip install kfp","lang":"bash","label":"Core SDK"},{"cmd":"pip install kfp[kubernetes]","lang":"bash","label":"SDK with Kubernetes Extras"}],"dependencies":[{"reason":"Required for defining pipeline specifications.","package":"kfp-pipeline-spec","optional":false},{"reason":"Required for interacting with the KFP API server.","package":"kfp-server-api","optional":false},{"reason":"Provides Kubernetes-specific features for pipeline authoring.","package":"kfp-kubernetes","optional":false}],"imports":[{"note":"The Client class for interacting with the KFP backend API is exposed directly under the 'kfp' namespace for convenience, though its canonical path is 'kfp.client.Client'.","symbol":"Client","correct":"from kfp import Client"},{"note":"For KFP SDK v2, DSL components are directly under the 'kfp.dsl' namespace. The 'kfp.v2' namespace for DSL was part of the v2-compatible mode and is now deprecated.","wrong":"from kfp.v2 import dsl","symbol":"dsl","correct":"from kfp import dsl"},{"symbol":"component","correct":"from kfp.dsl import component"},{"symbol":"pipeline","correct":"from kfp.dsl import pipeline"}],"quickstart":{"code":"import kfp\nfrom kfp import dsl\nimport os\n\n# Define a lightweight Python component\n@dsl.component\ndef add(a: float, b: float) -> float:\n    '''Calculates sum of two arguments'''\n    return a + b\n\n# Define a pipeline using the component\n@dsl.pipeline(\n    name='Addition pipeline',\n    description='An example pipeline that performs addition calculations.'\n)\ndef add_pipeline(\n    a: float = 1.0,\n    b: float = 7.0,\n):\n    first_add_task = add(a=a, b=4.0)\n    second_add_task = add(a=first_add_task.output, b=b)\n\n# --- Running the pipeline (requires KFP backend) ---\n# In a real environment, you'd configure the KFP client to connect to your KFP instance.\n# For local testing without a KFP backend, you can use `kfp.local.init`.\n\n# Example of compiling a pipeline (no KFP backend needed for this step)\n# compiler = kfp.compiler.Compiler()\n# compiler.compile(pipeline_func=add_pipeline, package_path='add_pipeline.yaml')\n\n# Example of running a pipeline against a KFP endpoint\n# client = kfp.Client(host=os.environ.get('KFP_HOST', 'http://localhost:8080'))\n# run = client.create_run_from_pipeline_func(\n#     add_pipeline,\n#     arguments={'a': 7.0, 'b': 8.0}\n# )\n# print(f\"Pipeline run initiated: {run.url}\")\n\nprint(\"Pipeline 'add_pipeline' defined successfully. To run, compile and submit to a KFP backend.\")","lang":"python","description":"This quickstart defines a simple Python function as a KFP component using `@dsl.component` and then orchestrates it into a pipeline using `@dsl.pipeline`. It demonstrates the basic structure for authoring KFP v2 pipelines. To actually execute the pipeline, it needs to be compiled and submitted to a running Kubeflow Pipelines backend."},"warnings":[{"fix":"Rewrite pipeline and component definitions to use the new KFP SDK v2 decorators and API patterns. Refer to the official migration guides.","message":"KFP SDK v2 is generally not backward compatible with user code written using the KFP SDK v1 main namespace. Key breaking changes include a new more Pythonic SDK with decorators like `@dsl.pipeline` and `@dsl.component`, and compilation to a generic Intermediate Representation (IR) YAML instead of Argo Workflow YAML.","severity":"breaking","affected_versions":"All versions migrating from KFP SDK v1 to v2 (starting with KFP v2.0.0)."},{"fix":"Review deployment configurations and explicitly specify MinIO manifests if you wish to continue using it, or migrate to SeaweedFS or another S3-compatible store.","message":"As of KFP 2.15.0, the default object store deployment for Kubeflow Pipelines has changed from MinIO to SeaweedFS. While MinIO is still supported, users upgrading from versions prior to 2.15.0 with existing or custom MinIO configurations for their backend may need to adjust their deployment manifests to maintain their desired object store configuration.","severity":"breaking","affected_versions":">=2.15.0"},{"fix":"Perform a full backup of your KFP database before upgrading to KFP 2.15.0 or later to ensure data recovery in case of issues with the migration.","message":"KFP 2.15.0 introduced a major upgrade to the underlying Gorm backend, necessitating an automated database index migration. This migration does not support rollback. It is strongly advised to back up production databases before initiating an upgrade from versions prior to 2.15.0.","severity":"breaking","affected_versions":">=2.15.0"},{"fix":"Check for official patch releases (e.g., 2.15.1, 2.15.2, etc.) that address this specific IRSA regression. If not patched, ensure these environment variables are correctly configured or fall back to an earlier stable version or an alternative authentication method if IRSA is critical.","message":"In KFP 2.15.0, a regression was identified for AWS S3 authentication using IAM Roles for Service Accounts (IRSA). Specifically, the environment variables `OBJECTSTORECONFIG_ACCESSKEY` and `OBJECTSTORECONFIG_SECRETACCESSKEY` (which could previously be empty or omitted when using IRSA) became implicitly required, leading to authentication failures.","severity":"gotcha","affected_versions":"2.15.0"}],"env_vars":null,"last_verified":"2026-04-08T00:00:00.000Z","next_check":"2026-07-07T00:00:00.000Z"}