{"id":5169,"library":"dagster-celery-k8s","title":"Dagster Celery Kubernetes","description":"dagster-celery-k8s is a Dagster integration library that enables scalable, distributed execution of Dagster jobs using Celery as a task queue and Kubernetes for orchestrating Celery workers and individual step execution. It allows users to leverage Celery for concurrency control and task isolation, while Kubernetes handles the underlying infrastructure for distributed execution. The current version is 0.29.0, with releases typically synchronized with the Dagster core library's rapid development cadence.","status":"active","version":"0.29.0","language":"en","source_language":"en","source_url":"https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-celery-k8s","tags":["dagster","celery","kubernetes","k8s","executor","orchestration","distributed"],"install":[{"cmd":"pip install dagster-celery-k8s","lang":"bash","label":"Install latest version"}],"dependencies":[{"reason":"Core Dagster library, required for defining and running pipelines.","package":"dagster"},{"reason":"Kubernetes integration for Dagster, provides core K8s run launcher and utilities.","package":"dagster-k8s"},{"reason":"Distributed task queue for managing and executing operations.","package":"celery"},{"reason":"Python client for Kubernetes API interaction.","package":"kubernetes"},{"reason":"Common Celery message broker (or 'redis')","package":"rabbitmq","optional":true},{"reason":"Common Celery message broker/backend (or 'rabbitmq')","package":"redis","optional":true}],"imports":[{"note":"Used in dagster.yaml to configure the run launcher.","symbol":"CeleryK8sRunLauncher","correct":"from dagster_k8s.launcher import CeleryK8sRunLauncher"},{"note":"Used in Python job definition to specify the executor.","symbol":"celery_k8s_job_executor","correct":"from dagster_celery_k8s.executor import celery_k8s_job_executor"}],"quickstart":{"code":"import os\nfrom dagster import job, op\nfrom dagster_celery_k8s.executor import celery_k8s_job_executor\n\n@op\ndef my_k8s_celery_op(context):\n    context.log.info(f\"Executing op on Celery K8s worker with PID {os.getpid()}\")\n    return \"Hello, Celery K8s!\"\n\n@job(executor_def=celery_k8s_job_executor)\ndef my_k8s_celery_job():\n    my_k8s_celery_op()\n\n# To configure this job, you would typically use a dagster.yaml like this:\n# run_launcher:\n#   module: dagster_k8s.launcher\n#   class: CeleryK8sRunLauncher\n#   config:\n#     instance_config_map: \"dagster-k8s-instance-config-map\"\n#     dagster_home: \"/opt/dagster/dagster_home\"\n#     broker: \"redis://\" + os.environ.get(\"REDIS_BROKER_HOST\", \"localhost\") + \":6379/0\"\n#     backend: \"redis://\" + os.environ.get(\"REDIS_BACKEND_HOST\", \"localhost\") + \":6379/1\"\n\n# And ensure Celery workers are running:\n# celery -A dagster_celery_k8s.app worker -l info --hostname=celery@%h","lang":"python","description":"This quickstart demonstrates a simple Dagster job configured to use the `celery_k8s_job_executor`. To run this, you would deploy Dagster to Kubernetes using its Helm chart, configure `CeleryK8sRunLauncher` in your `dagster.yaml` with appropriate broker and backend URLs (e.g., Redis or RabbitMQ), and ensure Celery workers are running with the `dagster_celery_k8s.app` module. The `dagster.yaml` configures the Dagster instance to use the `CeleryK8sRunLauncher`, which delegates step execution to Celery tasks that, in turn, launch Kubernetes Jobs for each step."},"warnings":[{"fix":"Update Celery worker command to `celery -A dagster_celery_k8s.app worker -l info` or similar, depending on your specific Celery module.","message":"The `dagster-celery` module was refactored into `dagster-celery`, `dagster-celery-k8s`, and `dagster-celery-docker` around Dagster version 0.9.0. If you are upgrading from older versions or migrating, ensure your Celery worker startup commands use the correct `-A` parameter, e.g., `celery -A dagster_celery_k8s.app worker -l info` for this library.","severity":"breaking","affected_versions":"<0.9.0"},{"fix":"Configure an appropriate persistent I/O manager (e.g., `s3_pickle_io_manager`, `gcs_pickle_io_manager`) in your Dagster job's resource definitions or instance configuration.","message":"When using `celery_k8s_job_executor`, ops exchange data between potentially different worker processes or nodes. This requires configuring a persistent I/O manager (e.g., S3, GCS, or an NFS mount) that is accessible by all Celery workers and step execution Kubernetes Jobs. Default in-memory or filesystem storage will not work in a distributed setup.","severity":"gotcha","affected_versions":"All"},{"fix":"Ensure `runLauncher.type` is set to `CeleryK8sRunLauncher` in your Helm `values.yaml` file.","message":"In Helm chart configurations for Dagster, the `runLauncher.type` field in `values.yaml` is case-sensitive and must be `CeleryK8sRunLauncher` (with proper capitalization). Incorrect capitalization will lead to validation errors during deployment.","severity":"gotcha","affected_versions":"All"},{"fix":"Verify that `broker` and `backend` URLs in `dagster.yaml` exactly correspond to the `CELERY_BROKER_URL` and `CELERY_RESULT_BACKEND` (or equivalent) settings used by your Celery workers.","message":"The Celery broker and backend URLs (e.g., for Redis or RabbitMQ) configured in your `dagster.yaml` for `CeleryK8sRunLauncher` and `celery_k8s_job_executor` must precisely match the configuration used to start your Celery workers. A mismatch will prevent workers from picking up tasks.","severity":"gotcha","affected_versions":"All"},{"fix":"Regularly consult the official Dagster changelog and upgrade guides before updating. Ensure your Python environment and other dependencies (like Pydantic) meet the latest combined requirements of your Dagster core and library versions.","message":"Dagster, and its libraries, are under active development. This means internal APIs and deployment configurations can change rapidly, sometimes leading to breaking changes or deprecations between minor or even patch versions. While `dagster-celery-k8s` itself has Python requirements `<3.15,>=3.10`, the core `dagster` library it depends on has its own evolving requirements (e.g., dropping Python 3.8 support and requiring `pydantic>=2` from Dagster 1.12.x).","severity":"gotcha","affected_versions":"All"}],"env_vars":null,"last_verified":"2026-04-13T00:00:00.000Z","next_check":"2026-07-12T00:00:00.000Z"}