Dagster Kubernetes Integration

0.29.0 · active · verified Fri Apr 10

The `dagster-k8s` library provides a robust integration for running Dagster with Kubernetes. It enables launching Dagster runs as Kubernetes Jobs, executing external code in Kubernetes pods directly from assets and ops using `PipesK8sClient`, and forms the foundation for Dagster's official Helm chart deployments. The library is actively maintained, with version `0.29.0` typically aligning with major releases of the core Dagster library.

Warnings

deprecated The `k8s_job_op` API has been superseded by `PipesK8sClient` for launching Kubernetes jobs from within Dagster ops. While `k8s_job_op` may still function, it is no longer the recommended approach.
Fix: Migrate usage from `k8s_job_op` to `PipesK8sClient` for launching Kubernetes pods from assets and ops.
gotcha Kubernetes configuration via `dagster-k8s/config` tags has different propagation rules depending on the executor. When using `k8s_job_executor` (which runs each step in its own pod), job-level `dagster-k8s/config` tags are NOT propagated to individual step pods.
Fix: For `k8s_job_executor`, use the `step_k8s_config` field directly on the executor definition to apply configuration to every step pod, or apply `dagster-k8s/config` tags to individual ops if fine-grained control is needed.
gotcha When deploying Dagster projects to Kubernetes, it is crucial to build and push a custom Docker image that contains your Dagster project code and all its Python dependencies, including `dagster-k8s`.
Fix: Ensure your deployment workflow includes a Dockerfile that copies your project, installs `dagster`, `dagster-k8s`, and any other project dependencies, and then build and push this image to an accessible container registry.
gotcha Some APIs within `dagster-k8s` may be marked as 'beta' and can introduce breaking changes in minor version releases.
Fix: Refer to the official Dagster documentation and release notes before upgrading minor versions to identify potential breaking changes in beta APIs and adjust your code accordingly.

Install

pip install dagster-k8s Install dagster-k8s

Imports

K8sRunLauncher
```
from dagster_k8s.launcher import K8sRunLauncher
```
Used for configuring the Dagster instance to launch runs as Kubernetes Jobs.
k8s_job_executor
```
from dagster_k8s import k8s_job_executor
```
An executor that launches each step of a Dagster job as a separate Kubernetes Job.
PipesK8sClient
```
from dagster_k8s import PipesK8sClient
```
The `k8s_job_op` API is superseded; `PipesK8sClient` is the recommended resource for launching Kubernetes pods directly from Dagster assets and ops.

Quickstart

This example demonstrates how to use `PipesK8sClient` to launch a Kubernetes pod from within a Dagster asset. The `PipesK8sClient` allows Dagster to receive real-time events from the initiated jobs running in Kubernetes. You would typically deploy this with a Dagster instance configured to run on Kubernetes.

import dagster as dg
from dagster_k8s import PipesK8sClient

@dg.asset
def k8s_pipes_asset(
    context: dg.AssetExecutionContext,
    k8s_pipes_client: PipesK8sClient
):
    # Replace 'pipes-example:v1' with your actual Docker image
    # The image must contain the necessary code and dependencies to execute the external process
    return k8s_pipes_client.run(
        context=context,
        image='pipes-example:v1',
        command=["python", "-c", "print('Hello from K8s pod!')"]
    ).get_materialize_result()

defs = dg.Definitions(
    assets=[k8s_pipes_asset],
    resources={
        "k8s_pipes_client": PipesK8sClient()
    },
)

view raw JSON →