{"id":8922,"library":"dagster-pandera","title":"Dagster Pandera Integration","description":"dagster-pandera provides an integration layer to use Pandera for data validation within Dagster data pipelines. It allows defining Pandera schemas for Dagster assets, ensuring data quality and correctness before data is consumed by downstream assets. The current version is 0.29.0, released in sync with Dagster core, which has a frequent release cadence, often multiple patch releases per week.","status":"active","version":"0.29.0","language":"en","source_language":"en","source_url":"https://github.com/dagster-io/dagster","tags":["dagster","pandera","data quality","data validation","etl","orchestration","python"],"install":[{"cmd":"pip install dagster-pandera","lang":"bash","label":"Install dagster-pandera"}],"dependencies":[{"reason":"Core Dagster library, version tightly coupled.","package":"dagster","optional":false},{"reason":"Data validation library used for schema definitions.","package":"pandera","optional":false}],"imports":[{"symbol":"pandera_schema_asset","correct":"from dagster_pandera import pandera_schema_asset"},{"note":"PanderaSchema is directly exposed at the top level of the dagster_pandera package since recent versions.","wrong":"from dagster_pandera.schema import PanderaSchema","symbol":"PanderaSchema","correct":"from dagster_pandera import PanderaSchema"}],"quickstart":{"code":"import pandas as pd\nimport pandera as pa\nfrom dagster import Definitions, asset, ScheduleDefinition, JobDefinition\nfrom dagster_pandera import pandera_schema_asset\n\n# 1. Define a Pandera DataFrameSchema\nmy_dataframe_schema = pa.DataFrameSchema(\n    columns={\n        \"id\": pa.Column(int, pa.Check.ge(0)),\n        \"name\": pa.Column(str, pa.Check.str_length(min_value=1)),\n        \"value\": pa.Column(float)\n    },\n    strict=True, # Ensure no extra columns are present\n    index=pa.Index(int, name=\"record_index\")\n)\n\n# 2. Define a Dagster asset using the pandera_schema_asset decorator\n@pandera_schema_asset(schema=my_dataframe_schema)\ndef validated_data() -> pd.DataFrame:\n    \"\"\"\n    An asset that produces a DataFrame and validates it against my_dataframe_schema.\n    If validation fails, the asset materialization will error.\n    \"\"\"\n    # Simulate data production\n    data = {\n        \"id\": [1, 2, 3],\n        \"name\": [\"Alice\", \"Bob\", \"Charlie\"],\n        \"value\": [10.1, 20.2, 30.3]\n    }\n    df = pd.DataFrame(data, index=[100, 101, 102])\n    return df\n\n@asset\ndef downstream_asset(validated_data: pd.DataFrame):\n    \"\"\"\n    This asset consumes the validated data, guaranteed to conform to the schema.\n    \"\"\"\n    print(f\"Downstream asset received validated data with {len(validated_data)} rows.\")\n    # Further processing with the validated DataFrame\n    return validated_data['value'].sum()\n\n# 3. Define the Dagster repository with assets and an example job\nmy_job = JobDefinition(name=\"my_validation_pipeline\", assets=[validated_data, downstream_asset])\n\ndefs = Definitions(\n    jobs=[my_job],\n    schedules=[\n        ScheduleDefinition(\n            job=my_job,\n            cron_schedule=\"0 0 * * *\", # Run daily at midnight\n            name=\"daily_validation_schedule\"\n        )\n    ]\n)","lang":"python","description":"This quickstart defines a `pandera.DataFrameSchema` and then uses the `pandera_schema_asset` decorator to apply this schema to a Dagster asset. If the DataFrame returned by `validated_data` does not conform to `my_dataframe_schema`, the asset materialization will fail, preventing invalid data from proceeding to `downstream_asset`. The example also shows how to define a simple Dagster job and schedule."},"warnings":[{"fix":"Always upgrade `dagster` and all `dagster-*` libraries in unison to their latest compatible versions. Use `pip install dagster dagster-pandera --upgrade` or ensure versions align in your `requirements.txt`.","message":"Version Mismatch with Dagster Core: `dagster-pandera`'s version is tightly coupled with `dagster` core's minor and patch versions. Upgrading `dagster` without also upgrading `dagster-pandera` (or vice-versa) can lead to unexpected behavior, `ImportError`s, or runtime crashes.","severity":"breaking","affected_versions":"<0.29.0"},{"fix":"When defining `pa.DataFrameSchema`, include `strict=True` to prevent unexpected extra columns from passing validation: `pa.DataFrameSchema(..., strict=True)`.","message":"Schema Strictness: By default, `pandera.DataFrameSchema` might not enforce that *only* the specified columns are present (i.e., it might allow extra columns). If you need to strictly enforce the column set, you must explicitly set `strict=True` in your schema definition.","severity":"gotcha","affected_versions":"All versions"},{"fix":"For extremely large datasets, consider strategies like sampling, partial validation, or optimizing your Pandera checks. Ensure adequate compute resources are allocated for validation steps.","message":"Performance with Large Datasets: Validating very large DataFrames (e.g., millions of rows) with complex Pandera schemas can be CPU and memory intensive, potentially increasing asset run times. Consider the impact on your pipeline performance.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-04-16T00:00:00.000Z","next_check":"2026-07-15T00:00:00.000Z","problems":[{"fix":"Install the package: `pip install dagster-pandera`","cause":"The `dagster-pandera` package is not installed in the Python environment where Dagster is running.","error":"ModuleNotFoundError: No module named 'dagster_pandera'"},{"fix":"Inspect the detailed error message to identify which part of the DataFrame violates the schema. Adjust your data generation logic or refine your `pandera.DataFrameSchema` definition.","cause":"The DataFrame produced by the asset does not conform to the `pandera.DataFrameSchema` defined for `pandera_schema_asset`. This error message is followed by details about which column/check failed.","error":"pandera.errors.SchemaError: Error while validating dataframe:"},{"fix":"Ensure that the function decorated with `pandera_schema_asset` always returns a `pandas.DataFrame` object, and not `None`, especially in error or edge cases. Add checks in your asset's code to handle potential `None` or empty data scenarios gracefully before returning.","cause":"This error often occurs within Pandera validation when the input to the schema is unexpectedly `None` instead of a DataFrame, or when a DataFrame operation inside a check receives `None`.","error":"TypeError: object of type 'NoneType' has no len()"}]}