Dagster Docker

0.29.0 · active · verified Sat Apr 11

Dagster-docker is a Dagster integration that enables launching and executing Dagster runs or individual ops/assets within Docker containers. As part of the broader Dagster ecosystem, it follows a separate versioning scheme (currently 0.29.0) but is released in lockstep with the core `dagster` library, which sees frequent updates (typically minor releases weekly or bi-weekly, with major changes less often).

Warnings

gotcha The machine running the Dagster daemon/webserver (for `DockerRunLauncher`) or the machine launching jobs (for `docker_executor`) must have access to a running Docker daemon.
Fix: Ensure Docker is installed and running, and the Dagster user has permissions to interact with the Docker socket (e.g., `sudo usermod -aG docker $USER` on Linux systems).
gotcha Docker containers launched by `dagster-docker` need to be able to communicate back to the Dagster API server (e.g., Dagit, Dagster daemon). Incorrect network configuration can lead to runs hanging or failing to report status.
Fix: Carefully configure the `network` or `network_mode` parameters in `container_kwargs`, ensuring that the container can reach the Dagster API server's host and port. Often, using a custom Docker network or host networking mode (if applicable) is required for seamless communication.
breaking `dagster-docker` versions are tightly coupled with the `dagster` core library. Using `dagster-docker` with a significantly mismatched `dagster` core version (e.g., `dagster-docker==0.29.0` with `dagster==1.0.0`) can lead to runtime errors or unexpected behavior due to API changes.
Fix: Always install `dagster-docker` alongside the corresponding version of `dagster` you are using. For `dagster-docker==0.29.0`, this means installing `dagster==1.13.0` (or a compatible `1.x.x` version). For example: `pip install dagster==1.13.0 dagster-docker==0.29.0`.

Install

pip install dagster-docker Install latest version

Imports

DockerRunLauncher
```
from dagster_docker import DockerRunLauncher
```
Primarily configured in `dagster.yaml` for a Dagster deployment to launch runs in Docker.
docker_executor
```
from dagster_docker import docker_executor
```
A pre-configured executor definition used when defining Dagster jobs programmatically to run ops/assets in Docker.

Quickstart

This quickstart demonstrates how to define a Dagster job that utilizes `dagster-docker`'s `docker_executor`. When this job is launched within a Dagster deployment, the `my_docker_asset` will execute inside a Docker container configured as specified. For this to run successfully, a Docker daemon must be accessible, and the specified Docker image should contain Python and any necessary dependencies for your asset code. Remember to replace `python:3.10-slim-buster` with an image containing your Dagster code for real-world scenarios.

from dagster import Definitions, asset, define_asset_job
from dagster_docker import docker_executor
import os

# Define a simple asset
@asset
def my_docker_asset():
    """An asset whose execution will be managed by a Docker executor."""
    message = os.environ.get("GREETING", "Hello from default asset!")
    print(message)
    return message

# Define a job that uses the docker_executor.
# The configuration demonstrates how to specify Docker image and environment variables
# for the container in which the asset will run.
my_docker_job = define_asset_job(
    name="my_docker_job",
    selection=[my_docker_asset],
    executor_def=docker_executor.configured(
        {
            "container_kwargs": {
                # This image needs to have Python and your Dagster code installed.
                # For a minimal example, we use a basic Python image. In production,
                # you'd build an image with your specific Dagster code and dependencies.
                "image": os.environ.get("DOCKER_IMAGE", "python:3.10-slim-buster"),
                "environment": {
                    "GREETING": "Hello from inside the Docker container!"
                },
                # Optional: specify a Docker network if your Dagster instance
                # is also running in Docker and needs to communicate.
                "network_mode": "bridge"
            },
            # If you want to pass host environment variables directly to the container,
            # list them here. The values will be taken from the environment where Dagster runs.
            "env_vars": ["HOST_VAR"]
        }
    ),
)

# To make these definitions loadable by Dagster (e.g., `dagster dev -f my_file.py`)
# you would typically expose them via a Definitions object:
# defs = Definitions(
#     jobs=[my_docker_job],
# )

view raw JSON →