{"id":3948,"library":"dagster-celery","title":"Dagster Celery Integration","description":"dagster-celery is a library that allows Dagster to use Celery as its execution engine, offloading op computations to a distributed Celery cluster. The current version is 0.29.0, which corresponds to Dagster core version 1.13.0. This library typically releases in lockstep with the main Dagster project, following a rapid, iterative release cadence.","status":"active","version":"0.29.0","language":"en","source_language":"en","source_url":"https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-celery","tags":["dagster","celery","orchestration","executor","distributed"],"install":[{"cmd":"pip install dagster-celery","lang":"bash","label":"Install dagster-celery"}],"dependencies":[{"reason":"Core Dagster library, required for defining jobs and ops, which dagster-celery executes.","package":"dagster"}],"imports":[{"note":"Main executor for running Dagster ops on a Celery cluster.","symbol":"celery_executor","correct":"from dagster_celery import celery_executor"},{"note":"Specialized executor for running Dagster ops on a Celery cluster within Kubernetes.","symbol":"celery_k8s_executor","correct":"from dagster_celery import celery_k8s_executor"}],"quickstart":{"code":"import os\nfrom dagster import Definitions, job, op\nfrom dagster_celery import celery_executor\n\n# Configure Celery broker and backend URLs via environment variables\n# Example for local development with Redis: export CELERY_BROKER_URL=\"redis://localhost:6379/0\"\n# Example for local development with Redis: export CELERY_RESULT_BACKEND=\"redis://localhost:6379/0\"\ncelery_broker_url = os.environ.get('CELERY_BROKER_URL', 'redis://localhost:6379/0')\ncelery_result_backend = os.environ.get('CELERY_RESULT_BACKEND', 'redis://localhost:6379/0')\n\ncelery_config = {\n    \"broker_url\": celery_broker_url,\n    \"result_backend\": celery_result_backend,\n    \"task_default_queue\": \"dagster_celery_queue\", # Optional: dedicated queue for Dagster tasks\n}\n\n@op\ndef my_celery_op(context):\n    context.log.info(\"Executing op on Celery worker!\")\n    return \"Hello from Celery!\"\n\n@job(executor_def=celery_executor.configured(celery_config))\ndef my_celery_job():\n    my_celery_op()\n\n# To run this, you would need to:\n# 1. Start a Redis server (or other message broker/backend).\n# 2. Start a Celery worker: celery -A dagster_celery.app worker -l info -Q dagster_celery_queue\n#    (Assuming your Dagster code is in a module named 'dagster_celery.app' or similar)\n# 3. Load the repository with 'dagster dev' and launch a run for 'my_celery_job'.\n\ndefs = Definitions(\n    jobs=[my_celery_job]\n)\n","lang":"python","description":"This quickstart demonstrates how to configure a Dagster job to use the Celery executor. It defines a simple op and a job, then configures `celery_executor` with basic broker and backend URLs (using environment variables for flexibility). For this to run, you need a running Celery broker/backend (e.g., Redis) and a Celery worker started with `celery -A your_module worker -l info` pointing to your Dagster code. The job will then submit its op execution to the Celery queue."},"warnings":[{"fix":"Ensure your Celery broker and result backend URLs are correct and accessible from both Dagster and Celery workers. Verify Celery workers are running and configured to listen on the correct queues (e.g., `task_default_queue`). Use `celery inspect ping` to check worker connectivity.","message":"The `dagster-celery` library requires an externally provisioned and managed Celery cluster (broker, backend, and workers). Dagster itself does not manage the lifecycle of Celery components. Misconfiguration of `broker_url` or `result_backend` in `celery_config`, or failure to start Celery workers, are common sources of errors.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Always use the `executor_def=celery_executor.configured(celery_config)` pattern when defining your jobs or `Definitions` object. Refer to the official Dagster documentation for the specific version you are using.","message":"Dagster's execution engine configuration has evolved across major versions. Older patterns involving `config_schema` or directly passing configuration dictionaries to `executor_def` might be deprecated or behave differently. Ensure you are using the `executor_def.configured(config)` pattern.","severity":"breaking","affected_versions":"Before 1.0.0, and some minor versions after."},{"fix":"Always install `dagster-celery` with a version that matches your `dagster` core version's major and minor numbers. For example, if `dagster` is `1.13.0`, `dagster-celery` should be `0.29.0` (as per the release notes, library versions are `core_minor_version - 100` for the major part and `core_patch_version` for the minor part after Dagster 1.0).","message":"Version compatibility between `dagster` core and `dagster-celery` is critical. Mismatched major versions can lead to runtime errors or unexpected behavior due to API changes in the core execution engine.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Thoroughly review the `dagster-celery` Kubernetes documentation. Ensure your Celery worker deployment correctly references the image containing your Dagster code, has network access to the broker/backend, and possesses necessary permissions and resources within Kubernetes.","message":"When using `celery_k8s_executor`, the Kubernetes cluster must be properly configured for Celery workers, including appropriate images, resource requests/limits, environment variables, secrets (for broker/backend), and persistent storage if needed. Issues with any of these can prevent ops from running or reporting results.","severity":"gotcha","affected_versions":"All versions using `celery_k8s_executor`"}],"env_vars":null,"last_verified":"2026-04-11T00:00:00.000Z","next_check":"2026-07-10T00:00:00.000Z"}