{"id":1859,"library":"prefect-aws","title":"Prefect AWS Integration","description":"Prefect AWS is an integration library for Prefect, a data workflow orchestration and observability platform. It provides battle-tested blocks, tasks, and infrastructure components for seamless interaction with various Amazon Web Services, including ECS, S3, Secrets Manager, Lambda, Batch, and Glue. The current version is 0.7.7 and it receives frequent updates, often aligning with the Prefect core library's release cycle. [1, 4, 24]","status":"active","version":"0.7.7","language":"en","source_language":"en","source_url":"https://github.com/PrefectHQ/prefect/tree/main/src/integrations/prefect-aws","tags":["prefect","aws","cloud","orchestration","workflow","etl","data-pipelines","s3","ecs","secrets-manager","lambda"],"install":[{"cmd":"pip install prefect-aws","lang":"bash","label":"Install prefect-aws"}],"dependencies":[{"reason":"Core Prefect library is required for orchestration and workflow definition. Version 0.7.7 of prefect-aws requires prefect>=3.6.24.","package":"prefect","optional":false},{"reason":"Underlying AWS SDK for Python. Although abstracted by prefect-aws, it's fundamental for AWS interactions.","package":"boto3","optional":false}],"imports":[{"note":"AwsCredentials block is directly under the top-level `prefect_aws` package, not a submodule like `credentials`.","wrong":"from prefect_aws.credentials import AwsCredentials","symbol":"AwsCredentials","correct":"from prefect_aws import AwsCredentials"},{"symbol":"S3Bucket","correct":"from prefect_aws.s3 import S3Bucket"},{"symbol":"s3_download","correct":"from prefect_aws.s3 import s3_download"},{"note":"`AwsSecret` was used in older versions/examples, `SecretsManager` is the current block for AWS Secrets Manager. [1, 8, 12]","wrong":"from prefect_aws import AwsSecret","symbol":"SecretsManager","correct":"from prefect_aws import SecretsManager"}],"quickstart":{"code":"import os\nfrom prefect import flow, task\nfrom prefect_aws import AwsCredentials\nfrom prefect_aws.s3 import S3Bucket, s3_upload, s3_download\n\n# NOTE: For a real application, create and save the AwsCredentials block in the Prefect UI\n# or programmatically (as shown below, but typically saved once).\n# os.environ.get is used here to prevent hardcoding sensitive credentials.\n\n@flow\ndef create_and_save_aws_credentials_block():\n    \"\"\"Example flow to create and save an AwsCredentials block.\"\"\"\n    aws_credentials = AwsCredentials(\n        aws_access_key_id=os.environ.get('AWS_ACCESS_KEY_ID', 'YOUR_ACCESS_KEY_ID'),\n        aws_secret_access_key=os.environ.get('AWS_SECRET_ACCESS_KEY', 'YOUR_SECRET_ACCESS_KEY'),\n        region_name=os.environ.get('AWS_REGION', 'us-east-1')\n    )\n    # Replace 'my-aws-credentials' with your desired block name\n    aws_credentials.save('my-aws-credentials', overwrite=True)\n    print(\"AwsCredentials block 'my-aws-credentials' saved.\")\n\n@task\nasync def upload_file_to_s3(bucket_name: str, key: str, content: str):\n    aws_credentials_block = await AwsCredentials.load('my-aws-credentials')\n    await s3_upload(\n        bucket=bucket_name,\n        key=key,\n        text=content,\n        aws_credentials=aws_credentials_block\n    )\n    print(f\"Uploaded '{key}' to s3://{bucket_name}\")\n\n@task\nasync def download_file_from_s3(bucket_name: str, key: str) -> str:\n    aws_credentials_block = await AwsCredentials.load('my-aws-credentials')\n    downloaded_content = await s3_download(\n        bucket=bucket_name,\n        key=key,\n        aws_credentials=aws_credentials_block\n    )\n    print(f\"Downloaded '{key}' from s3://{bucket_name}\")\n    return downloaded_content\n\n@flow\nasync def s3_interaction_flow(bucket_name: str = \"your-test-bucket\"): # Replace with a real bucket name\n    test_key = \"prefect-test-file.txt\"\n    test_content = \"Hello from Prefect AWS!\"\n\n    # Make sure credentials block exists\n    if not await AwsCredentials.exists('my-aws-credentials'):\n        print(\"AwsCredentials block not found. Run create_and_save_aws_credentials_block() first.\")\n        return\n\n    await upload_file_to_s3(bucket_name, test_key, test_content)\n    downloaded = await download_file_from_s3(bucket_name, test_key)\n    print(f\"Content verification: {downloaded == test_content}\")\n\nif __name__ == \"__main__\":\n    # First, ensure your AWS credentials block is saved (run this once or use Prefect UI)\n    # os.environ['AWS_ACCESS_KEY_ID'] = '...' # Set actual credentials\n    # os.environ['AWS_SECRET_ACCESS_KEY'] = '...' # Set actual credentials\n    # os.environ['AWS_REGION'] = '...' # Set actual region\n    create_and_save_aws_credentials_block()\n    \n    # Then run the S3 interaction flow (replace 'your-test-bucket' with an actual S3 bucket name)\n    import asyncio\n    asyncio.run(s3_interaction_flow(bucket_name='your-test-bucket'))\n","lang":"python","description":"This quickstart demonstrates how to create and save an `AwsCredentials` block, which is essential for authenticating with AWS services. It then shows how to use this block with `prefect-aws.s3` tasks to upload and download a file from an S3 bucket within a Prefect flow. Remember to replace placeholder values with actual AWS credentials and an existing S3 bucket name. For production, save credentials securely in Prefect Cloud/Server or via environment variables and IAM roles. [1, 4, 8, 13]"},"warnings":[{"fix":"Review the official Prefect migration guide from Prefect 1 to Prefect 2. Re-architect flows and deployments to align with Prefect 2 concepts, especially the use of blocks and workers. [29, 30]","message":"Prefect 2 (which `prefect-aws` is built for) introduced significant architectural changes from Prefect 1. Migrating from Prefect 1.x to Prefect 2.x requires updating flow definitions, deployment patterns (Agents vs. Workers), and infrastructure configurations. This is not a direct upgrade path. [19, 29, 30]","severity":"breaking","affected_versions":"All versions of `prefect-aws` (designed for Prefect 2+)"},{"fix":"Migrate from `ECSTask` blocks to using the Prefect ECS Worker and Work Pools. Refer to the Prefect documentation's upgrade guides for agents to workers and ECS worker deployment. [9, 11, 23]","message":"The `ECSTask` infrastructure block has been deprecated and is replaced by the ECS Worker. Users should migrate to the ECS Worker for enhanced functionality, scalability, and better performance. [23]","severity":"deprecated","affected_versions":"Versions up to `prefect-aws==0.5.x` (ECSTask deprecated in favor of ECS Worker from 0.6.x onwards)."},{"fix":"Always use `AwsCredentials` blocks, which can be securely saved in the Prefect UI, or leverage AWS IAM roles and environment variables for authentication. Avoid hardcoding sensitive information. [1, 4]","message":"Directly embedding AWS Access Key IDs and Secret Access Keys in flow code is insecure and not recommended for production. [1, 4, 12]","severity":"gotcha","affected_versions":"All versions"},{"fix":"Ensure your Python environment is running version 3.10 or newer. Upgrade Python if necessary.","message":"`prefect-aws` requires Python >=3.10 as of version 0.7.0. Older Prefect 2.x versions supported Python >=3.7, but this has been updated. [PyPI metadata, 24]","severity":"gotcha","affected_versions":"0.7.0 and later"},{"fix":"Use a dedicated virtual environment. If conflicts occur, inspect dependency trees (`pip install pipdeptree && pipdeptree -p prefect-aws`) and consider pinning specific versions of conflicting packages, especially `boto3` or `botocore`.","message":"Dependency conflicts can arise when installing `prefect-aws` alongside other Prefect extras or other libraries that heavily rely on `boto3`. [25]","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-04-09T00:00:00.000Z","next_check":"2026-07-08T00:00:00.000Z"}