Prefect Docker

0.7.1 · active · verified Sun Apr 12

The `prefect-docker` library provides Prefect integrations for orchestrating flow runs within Docker containers. It includes infrastructure blocks for defining Docker execution environments and workers for managing Docker-based work pools. This library is specifically designed for Prefect 2.x and later. The current version is 0.7.1, with releases typically aligning with new features or compatibility updates for the core Prefect library.

Warnings

breaking Prefect 1.x used `DockerExecutor`, while Prefect 2.x uses `DockerContainer` for infrastructure and `DockerWorker` for work pool agents. Code written for Prefect 1.x's Docker execution will not work with `prefect-docker`.
Fix: Rewrite execution configuration to use Prefect 2.x infrastructure concepts like `DockerContainer` blocks and `DockerWorker`s.
gotcha The Docker daemon must be running and accessible for `prefect-docker` to function. If the daemon is not active or the user lacks permissions, flow runs will fail to start.
Fix: Ensure Docker Desktop is running or the Docker service is active on your host. On Linux, ensure your user is part of the `docker` group (`sudo usermod -aG docker $USER`) and reboot or re-login.
gotcha Confusing `DockerContainer` (an infrastructure block) with `DockerWorker` (an agent that pulls work from a work pool). `DockerContainer` defines *how* a flow run executes; `DockerWorker` is the process that *starts* those `DockerContainer` instances.
Fix: `DockerContainer` should be used when defining a specific Docker environment for a deployment. `DockerWorker` should be used when running an agent to poll a work pool for runs configured to use Docker infrastructure.
gotcha Networking issues can prevent Docker containers from reaching the Prefect API server, especially if running the server and containers on different networks or hosts.
Fix: Ensure containers can resolve the Prefect API URL. This often involves configuring Docker container networks, passing `PREFECT_API_URL` correctly, or ensuring the API server is exposed on a public IP/hostname if not running locally.

Install

pip install prefect-docker Install prefect-docker

Imports

DockerContainer
```
from prefect_docker.infrastructure import DockerContainer
```
DockerContainer is part of the prefect-docker collection, not the core Prefect library directly. Ensure `prefect_docker` is imported.

DockerWorker

from prefect_docker.workers import DockerWorker

DockerContainerDeployment
```
from prefect_docker.deployments import DockerContainerDeployment
```
This specific deployment object is located in `prefect_docker.deployments`, not `prefect_docker.infrastructure`.

Quickstart

This quickstart demonstrates how to define a Prefect flow and configure it to run within a Docker container using the `DockerContainer` infrastructure block. It includes a runnable example that executes the flow locally using the specified Docker image. For full deployment, the infrastructure block would typically be saved and referenced in a Prefect deployment.

from prefect import flow
from prefect_docker.infrastructure import DockerContainer

@flow(log_prints=True)
def hello_docker_flow(name: str = "World"):
    """A simple flow that prints a greeting."""
    print(f"Hello, {name} from a Docker container!")

if __name__ == "__main__":
    # Define a DockerContainer infrastructure block.
    # This block specifies how your flow runs will be executed in Docker.
    docker_infra = DockerContainer(
        image="python:3.10-slim-buster", # Use a small, readily available image
        # You can also specify other Docker options like network, volumes, env, etc.
    )

    # For a quick local test without full Prefect deployment,
    # you can directly apply the infrastructure to the flow.
    # Ensure your Docker daemon is running.
    print("Running flow locally using DockerContainer infrastructure...")
    hello_docker_flow.with_options(infrastructure=docker_infra)("TestUser")
    print("Flow run completed.")

    # For full Prefect deployment, you would typically save the block:
    # import asyncio
    # asyncio.run(docker_infra.save("my-docker-infra-block", overwrite=True))
    # Then deploy via CLI: `prefect deploy --name "my-flow" --infra-block docker-container/my-docker-infra-block`

view raw JSON →