Prefect Shell

0.3.5 · active · verified Sun Apr 12

Prefect Shell provides integrations for executing shell commands within Prefect flows. It allows users to embed shell scripts and commands directly into their data pipelines, leveraging Prefect's orchestration capabilities like logging, retries, and observability. This library is part of the Prefect ecosystem, currently at version 0.3.5, and is actively maintained with regular updates as part of the broader Prefect project.

Warnings

breaking Prefect 2.x changed the core API from Prefect 1.x. `prefect-shell` is designed specifically for Prefect 2 and is not compatible with Prefect 1. Ensure your Prefect core installation is version 2.0 or higher.
Fix: Upgrade your Prefect core installation to Prefect 2.x (e.g., `pip install -U prefect`) or ensure your environment uses Prefect 2.x if migrating from an older Prefect 1.x project.
gotcha When using `ShellOperation`, calling `.run()` is suitable for short, synchronous commands. For long-running processes or when needing to manage the subprocess lifecycle (e.g., waiting for completion, capturing output), use the context manager pattern (`with ShellOperation(...)`) combined with `.trigger()` and `.wait_for_completion()`, followed by `.fetch_result()` to get output. Directly calling `.run()` inside a flow will execute synchronously and may block.
Fix: For synchronous, short commands and immediate results: `result = ShellOperation(commands=['command']).run()`. For asynchronous execution, explicit process management, or long-running tasks: `with ShellOperation(...) as op: process = op.trigger(); process.wait_for_completion(); output = process.fetch_result()`.
gotcha To use `ShellOperation` as a Prefect block type (e.g., for creating and managing shell commands directly in the Prefect UI), you must explicitly register the blocks after installation.
Fix: Run `prefect block register -m prefect_shell` in your terminal after installing the library to make its block types available in the Prefect UI.

Install

pip install "prefect[shell]" Recommended installation (includes Prefect)
pip install prefect-shell Install only prefect-shell

Imports

ShellOperation

from prefect_shell import ShellOperation

flow
```
from prefect import flow
```

Quickstart

This quickstart demonstrates how to define a Prefect flow that executes shell commands using `ShellOperation`. It shows both the `.run()` method for simple, short commands and the context manager pattern (`with ShellOperation(...) as op: ...`) for long-running operations or when explicit output capture is required.

from prefect import flow
from prefect_shell import ShellOperation

@flow
def run_simple_command():
    # For short-running operations, use the .run() method
    result = ShellOperation(commands=["echo Hello from Prefect Shell!"]).run()
    print(f"Command output: {result.stdout.strip()}")

    # For longer-running operations or capturing structured output, use a context manager
    with ShellOperation(
        commands=["ls -l /tmp"],
        working_dir="/tmp"
    ) as list_files_operation:
        process = list_files_operation.trigger()
        process.wait_for_completion()
        output_lines = process.fetch_result()
        print("\nFiles in /tmp:")
        for line in output_lines:
            print(line.strip())

if __name__ == "__main__":
    run_simple_command()

view raw JSON →