Dagster

1.12.22 · active · verified Thu Apr 09

Dagster is an orchestration platform designed for the development, production, and observation of data assets. It emphasizes defining pipelines as data assets and provides a rich development experience with built-in UI and testing capabilities. The library is actively maintained with frequent minor releases, often on a weekly basis, incorporating new features, bug fixes, and improvements.

Warnings

breaking Dagster 1.0.0 introduced a major API overhaul, deprecating `solid` and `pipeline` in favor of `op` and `job`/`graph` respectively. Code written for pre-1.0 versions is not compatible without migration.
Fix: Replace `@solid` with `@op` and `PipelineDefinition` with `JobDefinition` or `@job` decorator. Refer to the Dagster 1.0 migration guide for full details.
gotcha The Dagster UI (Dagit) is provided by the `dagster-webserver` package, which must be installed separately from `dagster` core for local development and observation.
Fix: Ensure both `dagster` and `dagster-webserver` are installed: `pip install dagster dagster-webserver`.
gotcha Starting with Dagster 1.12.18, `psycopg2-binary` is no longer a transitive dependency of `dagster-postgres`. If you use `dagster-postgres` for event log or run storage and rely on this dependency, your environment may break.
Fix: Explicitly add `psycopg2-binary` (or `psycopg2`) to your project's dependencies if you use `dagster-postgres`: `pip install psycopg2-binary`.
gotcha There are two main CLIs: `dagster` for open-source Dagster environments and `dg` (from `dagster-cloud`) for Dagster Cloud deployments. Using the wrong CLI can lead to confusion or errors.
Fix: For open-source Dagster, use `dagster dev`, `dagster job launch`, etc. For Dagster Cloud, install `dagster-cloud` and use `dg deploy`, `dg api`, etc.
deprecated Older methods of defining resource and op configuration using `config_schema` or directly passing dicts are deprecated. The modern approach uses Pydantic-style `Config` classes.
Fix: Define configuration using `class MyConfig(Config): ...` and type-hint your resource/op functions with it: `def my_resource(config: MyConfig): ...`.

Install

pip install dagster dagster-webserver Core and Web UI
pip install dagster-cloud Dagster Cloud CLI (optional)

Imports

asset
```
from dagster import asset
```
job
```
from dagster import job
```
op
```
from dagster import op
```
The `solid` decorator was replaced by `op` in Dagster 1.0.0.
graph
```
from dagster import graph
```
Definitions
```
from dagster import Definitions
```
Config
```
from dagster import Config
```
Modern Dagster uses Pydantic-style `Config` models for resource/op configuration instead of `config_schema`.
Schedule
```
from dagster import Schedule
```
Sensor
```
from dagster import Sensor
```

Quickstart

This quickstart defines a simple asset and a job that materializes it. It includes a directly runnable Python snippet to execute the asset in memory. For local development with the full Dagster UI (Dagit), save the commented `Definitions` block to a file and run `dagster dev`.

from dagster import Definitions, asset, job

@asset
def my_first_asset():
    """A simple asset that prints a message and returns a string."""
    print("Hello, Dagster!")
    return "hello"

@job
def my_asset_job():
    my_first_asset()

# To run this locally and see the output immediately:
if __name__ == "__main__":
    from dagster import materialize_to_memory
    result = materialize_to_memory(assets=[my_first_asset])
    assert result.success
    print(f"Asset output: {result.output_for_node('my_first_asset')}")

# For UI (Dagit) integration, save the following `Definitions` to a file (e.g., `my_repo.py`):
# defs = Definitions(
#     assets=[my_first_asset],
#     jobs=[my_asset_job],
# )
# Then run `dagster dev -f my_repo.py` and navigate to http://localhost:3000

view raw JSON →