{"id":2455,"library":"dagster-k8s","title":"Dagster Kubernetes Integration","description":"The `dagster-k8s` library provides a robust integration for running Dagster with Kubernetes. It enables launching Dagster runs as Kubernetes Jobs, executing external code in Kubernetes pods directly from assets and ops using `PipesK8sClient`, and forms the foundation for Dagster's official Helm chart deployments. The library is actively maintained, with version `0.29.0` typically aligning with major releases of the core Dagster library.","status":"active","version":"0.29.0","language":"en","source_language":"en","source_url":"https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-k8s","tags":["dagster","kubernetes","orchestration","etl","cloud-native","data-pipeline"],"install":[{"cmd":"pip install dagster-k8s","lang":"bash","label":"Install dagster-k8s"}],"dependencies":[{"reason":"Core Dagster library is required for defining and orchestrating data pipelines.","package":"dagster"},{"reason":"Commonly used for persistent metadata storage in Kubernetes deployments, as seen in Helm charts and Dockerfile examples.","package":"dagster-postgres","optional":true}],"imports":[{"note":"Used for configuring the Dagster instance to launch runs as Kubernetes Jobs.","symbol":"K8sRunLauncher","correct":"from dagster_k8s.launcher import K8sRunLauncher"},{"note":"An executor that launches each step of a Dagster job as a separate Kubernetes Job.","symbol":"k8s_job_executor","correct":"from dagster_k8s import k8s_job_executor"},{"note":"The `k8s_job_op` API is superseded; `PipesK8sClient` is the recommended resource for launching Kubernetes pods directly from Dagster assets and ops.","wrong":"from dagster_k8s import k8s_job_op","symbol":"PipesK8sClient","correct":"from dagster_k8s import PipesK8sClient"}],"quickstart":{"code":"import dagster as dg\nfrom dagster_k8s import PipesK8sClient\n\n@dg.asset\ndef k8s_pipes_asset(\n    context: dg.AssetExecutionContext,\n    k8s_pipes_client: PipesK8sClient\n):\n    # Replace 'pipes-example:v1' with your actual Docker image\n    # The image must contain the necessary code and dependencies to execute the external process\n    return k8s_pipes_client.run(\n        context=context,\n        image='pipes-example:v1',\n        command=[\"python\", \"-c\", \"print('Hello from K8s pod!')\"]\n    ).get_materialize_result()\n\ndefs = dg.Definitions(\n    assets=[k8s_pipes_asset],\n    resources={\n        \"k8s_pipes_client\": PipesK8sClient()\n    },\n)","lang":"python","description":"This example demonstrates how to use `PipesK8sClient` to launch a Kubernetes pod from within a Dagster asset. The `PipesK8sClient` allows Dagster to receive real-time events from the initiated jobs running in Kubernetes. You would typically deploy this with a Dagster instance configured to run on Kubernetes."},"warnings":[{"fix":"Migrate usage from `k8s_job_op` to `PipesK8sClient` for launching Kubernetes pods from assets and ops.","message":"The `k8s_job_op` API has been superseded by `PipesK8sClient` for launching Kubernetes jobs from within Dagster ops. While `k8s_job_op` may still function, it is no longer the recommended approach.","severity":"deprecated","affected_versions":"<=0.28.x"},{"fix":"For `k8s_job_executor`, use the `step_k8s_config` field directly on the executor definition to apply configuration to every step pod, or apply `dagster-k8s/config` tags to individual ops if fine-grained control is needed.","message":"Kubernetes configuration via `dagster-k8s/config` tags has different propagation rules depending on the executor. When using `k8s_job_executor` (which runs each step in its own pod), job-level `dagster-k8s/config` tags are NOT propagated to individual step pods.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Ensure your deployment workflow includes a Dockerfile that copies your project, installs `dagster`, `dagster-k8s`, and any other project dependencies, and then build and push this image to an accessible container registry.","message":"When deploying Dagster projects to Kubernetes, it is crucial to build and push a custom Docker image that contains your Dagster project code and all its Python dependencies, including `dagster-k8s`.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Refer to the official Dagster documentation and release notes before upgrading minor versions to identify potential breaking changes in beta APIs and adjust your code accordingly.","message":"Some APIs within `dagster-k8s` may be marked as 'beta' and can introduce breaking changes in minor version releases.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-04-10T00:00:00.000Z","next_check":"2026-07-09T00:00:00.000Z"}