{"id":7126,"library":"dagster-shell","title":"Dagster Shell","description":"dagster-shell is a Python package within the Dagster ecosystem, providing functionality to execute shell commands and scripts as operations within Dagster workflows. While the library itself is at version 0.25.13, its core functionality for creating shell ops has been deprecated in favor of a newer approach within the main `dagster` package. The broader Dagster platform maintains an active release cadence, with minor versions (potentially including breaking changes) approximately every 12 weeks and weekly patch releases for deprecations.","status":"active","version":"0.25.13","language":"en","source_language":"en","source_url":"https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-shell","tags":["dagster","etl","orchestration","shell","bash","subprocess"],"install":[{"cmd":"pip install dagster-shell","lang":"bash","label":"Install dagster-shell"}],"dependencies":[{"reason":"This library is an integration for the Dagster orchestration framework and requires `dagster` itself.","package":"dagster","optional":false},{"reason":"Requires Python versions between 3.9 and 3.12 (exclusive of 3.13).","package":"python","optional":false}],"imports":[{"note":"The `create_shell_command_op` and `create_shell_script_op` from `dagster-shell` are deprecated. Use `PipesSubprocessClient` from the `dagster` package instead.","wrong":"from dagster_shell import create_shell_command_op","symbol":"PipesSubprocessClient","correct":"from dagster import PipesSubprocessClient"}],"quickstart":{"code":"import dagster as dg\nimport os\n\n@dg.asset\ndef run_shell_command_asset(context: dg.AssetExecutionContext, pipes_subprocess_client: dg.PipesSubprocessClient):\n    # Example: Run a simple echo command\n    # The command is executed by PipesSubprocessClient, which pipes logs back to Dagster.\n    command = [\"bash\", \"-c\", \"echo 'Hello from Dagster Pipes!' && sleep 1 && echo 'Done!'\"]\n    \n    # For a shell script file, you'd specify its path:\n    # shell_script_path = \"./my_script.sh\"\n    # with open(shell_script_path, \"w\") as f:\n    #     f.write(\"#!/bin/bash\\nset -eux\\necho 'Executing my_script.sh'\\n\")\n    # command = [\"bash\", shell_script_path]\n\n    # Run the command and get results. PipesSubprocessClient streams logs and events.\n    result = pipes_subprocess_client.run(command=command, context=context).get_results()\n    \n    # The result object contains information about the subprocess execution, e.g., return code\n    context.log.info(f\"Shell command completed with exit code: {result.return_code}\")\n\ndefinitions = dg.Definitions(\n    assets=[run_shell_command_asset],\n    resources={\n        \"pipes_subprocess_client\": dg.PipesSubprocessClient(),\n        # pipes_subprocess_client can be configured, e.g., to pass environment variables\n        # \"pipes_subprocess_client\": dg.PipesSubprocessClient(env={'MY_VAR': 'my_value'})\n    },\n)\n","lang":"python","description":"This quickstart demonstrates how to use `PipesSubprocessClient` from the main `dagster` package to execute a shell command as a Dagster asset. This is the recommended approach as the direct shell ops in `dagster-shell` are deprecated. The `PipesSubprocessClient` handles streaming logs and events from the shell process back to Dagster."},"warnings":[{"fix":"Migrate to using `PipesSubprocessClient` from the `dagster` package for executing shell commands and scripts. See the quickstart example for correct usage.","message":"The `create_shell_command_op` and `create_shell_script_op` functions from `dagster-shell` have been deprecated in Dagster 1.10+ and should no longer be used.","severity":"breaking","affected_versions":"Dagster 1.10+"},{"fix":"Ensure you are using a recent version of Dagster and `PipesSubprocessClient` which correctly merges environments. If using older `dagster-shell` ops, explicitly merge `os.environ` with your desired `env` dictionary before passing it to the op config. Example: `{**os.environ, **my_custom_env}`.","message":"When specifying environment variables for shell commands, older versions of `dagster-shell` (e.g., pre-1.0 and some 0.15.x versions) might *override* the entire environment instead of merging with the parent process's environment.","severity":"gotcha","affected_versions":"<1.0 (Dagster core) and some 0.15.x versions"},{"fix":"Consider running Dagster environments on Linux-based systems. If Windows is required, ensure your shell commands are simple and do not rely on `preexec_fn` functionality, or explore alternative execution mechanisms.","message":"`dagster-shell` (and underlying Python `subprocess` calls) may encounter issues on Windows due to `preexec_fn` not being supported. This can manifest as `RuntimeError` or unexpected behavior.","severity":"gotcha","affected_versions":"All versions, specifically on Windows operating systems"},{"fix":"Ensure commands are correctly quoted and consider explicitly using `shlex.split` for complex commands. If issues persist with binary executables, confirm the executable path is correct and accessible, and that `dagster-shell` is not misinterpreting arguments due to encoding. Consider wrapping the binary call in a simple bash script.","message":"Executing binary files or commands with complex arguments via `dagster-shell` might lead to 'not-found' errors or unexpected behavior if UTF-8 encoding is implicitly applied during command interpretation.","severity":"gotcha","affected_versions":"Specific versions, e.g., Dagster 1.3.13 / dagster-shell 0.19.13"}],"env_vars":null,"last_verified":"2026-04-16T00:00:00.000Z","next_check":"2026-07-15T00:00:00.000Z","problems":[{"fix":"Update your code to use `PipesSubprocessClient` from the `dagster` package instead. The import should be `from dagster import PipesSubprocessClient`, and usage will involve configuring it as a resource and calling its `run` method.","cause":"Attempting to import deprecated functions `create_shell_command_op` or `create_shell_script_op` from the `dagster-shell` library after Dagster version 1.10. These functions have been removed.","error":"ImportError: cannot import name 'create_shell_command_op' from 'dagster_shell'"},{"fix":"Run your Dagster instance and code on a Linux-based environment or container. If a Windows environment is unavoidable, you may need to find alternative ways to execute shell commands that do not rely on `preexec_fn` or implement custom execution logic.","cause":"The underlying `subprocess` call used by `dagster-shell` attempts to use `preexec_fn`, a Unix-specific feature, on a Windows operating system.","error":"RuntimeError: preexec_fn is not supported on Windows platforms"},{"fix":"Verify that the executable or script is in the system's PATH or provide its absolute path. Ensure complex arguments are properly quoted. If the command involves a binary, consider wrapping it in a simple bash script to explicitly control execution and argument passing.","cause":"The shell command or binary executable could not be found, or its arguments were misinterpreted, possibly due to encoding issues or incorrect paths. Return code 127 usually indicates 'command not found'.","error":"Shell command execution failed with output: [output_logs] and return code: 127 (or other non-zero code), typically for binary executables or complex commands."}]}