dagster-polars

0.27.9 verified Fri May 01 auth: no python

Dagster integration library for Polars, providing IO managers, types, and utilities to read/write Polars DataFrames from/to various file formats (Parquet, CSV, Delta Lake, etc.). Current version: 0.27.9. Active development, follows Dagster releases.

pip install dagster-polars

Common errors

error ModuleNotFoundError: No module named 'dagster_polars.io_manager' ↓

cause Import path changed in version 0.19.0.

fix

Use from dagster_polars import PolarsIOManager.

error dagster._core.errors.DagsterTypeError: Type check failed for input 'my_asset' - expected type 'DataFrame', got '<class 'NoneType'>' ↓

cause Asset function returned None instead of a Polars DataFrame.

fix

Ensure asset returns a Polars DataFrame, e.g., return pl.DataFrame(...).

Warnings

breaking Version 0.19.0 changed the import path for PolarsIOManager from `dagster_polars.io_manager` to `dagster_polars`. Old imports will break. ↓

fix Update imports to `from dagster_polars import PolarsIOManager`.

gotcha PolarsIOManager by default uses the file extension to determine format. If your asset output doesn't have an explicit extension, it may fail or save as Parquet unexpectedly. ↓

fix Specify `extension` in the IO manager config or use `PolarsParquetIOManager` / `PolarsCSVIOManager` for explicit format.

gotcha When using with Dagster's `@asset` decorator, the asset function must return a Polars DataFrame. Returning None or other types may cause errors. ↓

fix Ensure assets return a `pl.DataFrame` or use `PolarsBaseIOManager` if you need custom behavior.

Imports

PolarsIOManager

wrong

from dagster_polars.io_manager import PolarsIOManager

correct

from dagster_polars import PolarsIOManager

Older submodule path removed in 0.19

PolarsParquetIOManager

from dagster_polars import PolarsParquetIOManager

Quickstart

Define an asset that returns a Polars DataFrame and use PolarsIOManager to persist it.

from dagster import Definitions, asset
from dagster_polars import PolarsIOManager
import polars as pl

@asset
def my_asset() -> pl.DataFrame:
    return pl.DataFrame({"x": [1, 2, 3]})

defs = Definitions(
    assets=[my_asset],
    resources={"io_manager": PolarsIOManager(base_dir="./data")}
)