{"id":1976,"library":"dagster-aws","title":"Dagster AWS","description":"Dagster-aws provides a collection of integrations for common AWS services, enabling Dagster to orchestrate workloads involving S3, ECS, Lambda, EMR, and more. It offers resources, run launchers, and IO managers to seamlessly connect Dagster assets and operations with your AWS infrastructure. The current version is 0.28.22, and it typically releases monthly, in conjunction with major Dagster core updates.","status":"active","version":"0.28.22","language":"en","source_language":"en","source_url":"https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-aws","tags":["dagster","aws","etl","orchestration","data-pipeline","s3","ecs","lambda"],"install":[{"cmd":"pip install dagster-aws","lang":"bash","label":"Install dagster-aws"}],"dependencies":[{"reason":"Core Dagster framework, required for all Dagster integrations.","package":"dagster","optional":false},{"reason":"AWS SDK for Python, used for interacting with AWS services.","package":"boto3","optional":false}],"imports":[{"symbol":"S3Resource","correct":"from dagster_aws.s3 import S3Resource"},{"symbol":"s3_io_manager","correct":"from dagster_aws.s3.io_manager import s3_io_manager"},{"symbol":"EcsRunLauncher","correct":"from dagster_aws.ecs import EcsRunLauncher"},{"symbol":"LambdaRunLauncher","correct":"from dagster_aws.lambda_libs import LambdaRunLauncher"},{"symbol":"emr_resource","correct":"from dagster_aws.emr import emr_resource"}],"quickstart":{"code":"import os\nfrom dagster import Definitions, asset, Config\nfrom dagster_aws.s3 import S3Resource\n\nclass MyS3Config(Config):\n    bucket: str\n    key: str\n\n@asset\ndef my_s3_asset(context, s3: S3Resource, config: MyS3Config):\n    \"\"\"\n    Writes a simple string to an S3 object.\n    \"\"\"\n    s3.get_client().put_object(\n        Bucket=config.bucket,\n        Key=config.key,\n        Body=\"Hello from Dagster S3!\"\n    )\n    context.log.info(f\"Wrote to s3://{config.bucket}/{config.key}\")\n\ndefs = Definitions(\n    assets=[my_s3_asset],\n    resources={\n        \"s3\": S3Resource(\n            region_name=os.environ.get(\"AWS_REGION\", \"us-east-1\"),\n            # For local testing, ensure these are set as env vars or use other AWS auth methods\n            aws_access_key_id=os.environ.get(\"AWS_ACCESS_KEY_ID\", \"\"),\n            aws_secret_access_key=os.environ.get(\"AWS_SECRET_ACCESS_KEY\", \"\")\n        )\n    }\n)\n\n# To run:\n# 1. Ensure AWS credentials and AWS_REGION are set in your environment variables.\n# 2. dagster dev -f your_file.py\n# 3. In the UI, launch a run for 'my_s3_asset' with a config like:\n#    {\"ops\": {\"my_s3_asset\": {\"inputs\": {\"config\": {\"bucket\": \"your-bucket-name\", \"key\": \"my-dagster-object.txt\"}}}}}}","lang":"python","description":"This quickstart demonstrates how to define a Dagster asset that interacts with Amazon S3 using `S3Resource`. It writes a simple string to a specified S3 bucket and key. Ensure you have AWS credentials configured in your environment or via IAM roles for this example to run successfully."},"warnings":[{"fix":"Consult the official Dagster documentation or `setup.py` of the respective library for precise version compatibility before installing or upgrading.","message":"Dagster library versions (e.g., `dagster-aws` 0.x.y) are tightly coupled to specific `dagster` core versions (e.g., 1.x.y). Always ensure you install compatible versions to avoid runtime errors; mismatching minor versions is a common source of issues. For `dagster-aws` 0.28.x, ensure `dagster` core is 1.12.x.","severity":"gotcha","affected_versions":"<=0.28.22"},{"fix":"Verify your AWS environment setup, IAM roles, and credential configuration. Use AWS CLI to test permissions for the specific services Dagster interacts with.","message":"All interactions with AWS services (S3, ECS, Lambda, EMR, etc.) require correct AWS credentials and IAM permissions. Ensure the underlying compute environment (e.g., EC2 instance, ECS task, EKS pod) has an appropriate IAM role attached, or explicitly configure credentials via environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`) or `~/.aws/credentials` for local development. Incorrect permissions lead to `AccessDenied` or `NoCredentialsError` exceptions.","severity":"gotcha","affected_versions":"<=0.28.22"},{"fix":"Pass the `region_name` parameter explicitly to your Dagster AWS resources, e.g., `S3Resource(region_name='us-west-2')`.","message":"AWS resources like `S3Resource` or `EMRResource` may infer the AWS region from environment variables (`AWS_REGION`) or your `~/.aws/config` file. If running across multiple regions or in non-standard environments (e.g., localstack), always explicitly configure the `region_name` parameter to avoid unexpected cross-region errors or latency.","severity":"gotcha","affected_versions":"<=0.28.22"}],"env_vars":null,"last_verified":"2026-04-09T00:00:00.000Z","next_check":"2026-07-08T00:00:00.000Z"}