{"id":8655,"library":"snakemake-storage-plugin-s3","title":"Snakemake S3 Storage Plugin","description":"The `snakemake-storage-plugin-s3` library provides a Snakemake storage plugin for interacting with S3-compatible object storage systems, including AWS S3, MinIO, and others. It allows Snakemake workflows to read and write files directly from/to S3 buckets using `s3://` URIs. The project is actively maintained with frequent bug fix releases, typically every few months, ensuring compatibility and robustness.","status":"active","version":"0.3.6","language":"en","source_language":"en","source_url":"https://github.com/snakemake/snakemake-storage-plugin-s3","tags":["snakemake","s3","storage","aws","minio","cloud"],"install":[{"cmd":"pip install snakemake-storage-plugin-s3","lang":"bash","label":"Install plugin"}],"dependencies":[{"reason":"This is a plugin for Snakemake and requires Snakemake to be installed.","package":"snakemake","optional":false},{"reason":"Used for interacting with the S3 API.","package":"boto3","optional":false},{"reason":"Provides the necessary interface for Snakemake storage plugins.","package":"snakemake-interface-storage-plugins","optional":false}],"imports":[{"note":"You do not typically import classes directly from this library into your Python code. Snakemake discovers and utilizes the plugin automatically when it's installed in the same environment.","symbol":"S3Storage","correct":"This plugin is automatically discovered by Snakemake when installed, and is used via 's3://' URIs in your Snakefile, not by direct Python import of a specific class for usage."}],"quickstart":{"code":"# file: Snakefile\n\n# Set S3 credentials via environment variables for a runnable example\n# export AWS_ACCESS_KEY_ID=\"YOUR_ACCESS_KEY\"\n# export AWS_SECRET_ACCESS_KEY=\"YOUR_SECRET_KEY\"\n# export AWS_ENDPOINT_URL=\"http://localhost:9000\" # Optional, for MinIO or other S3-compatible storages\n\nrule all:\n    input: \"s3://my-test-bucket/output.txt\"\n\nrule generate_s3_file:\n    output: \"s3://my-test-bucket/output.txt\"\n    shell:\n        \"echo 'Hello Snakemake S3!' > {output}\"\n\n# To run this example:\n# 1. Ensure snakemake and snakemake-storage-plugin-s3 are installed.\n# 2. Set your S3 credentials (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION, etc.)\n#    or AWS_ENDPOINT_URL for compatible S3 services.\n# 3. Create 'my-test-bucket' in your S3 service.\n# 4. Navigate to the directory containing this Snakefile and run: snakemake","lang":"python","description":"This quickstart demonstrates how to use the S3 storage plugin in a `Snakefile`. Snakemake automatically recognizes `s3://` prefixes for input and output files when the plugin is installed. Ensure your S3 credentials are configured in your environment or via standard `boto3` configuration methods (e.g., `~/.aws/credentials`). For S3-compatible services like MinIO, `AWS_ENDPOINT_URL` is typically required."},"warnings":[{"fix":"Ensure all S3 object paths used in your Snakefile are absolute or relative paths without `..` segments. For example, use `s3://bucket/dir/file.txt` instead of `s3://bucket/dir/../file.txt`.","message":"S3 object keys cannot contain `..` (double-dot) for path traversal. The plugin attempts to normalize paths, but directly using `..` in S3 URIs will lead to errors.","severity":"gotcha","affected_versions":"All versions prior to 0.3.5, and potentially current if complex paths are not normalized correctly."},{"fix":"Upgrade to version 0.3.3 or newer, which includes a workaround for this `boto3` behavior. If issues persist, consult `boto3` documentation or your S3 provider for specific configuration requirements regarding checksums.","message":"When using non-AWS S3 compatible services (e.g., MinIO), `boto3` might automatically attach checksums to uploaded files which can cause issues or unexpected behavior.","severity":"gotcha","affected_versions":"All versions prior to 0.3.3."},{"fix":"Always keep the `snakemake-storage-plugin-s3` library updated to its latest version to ensure compatibility with recent Snakemake releases and the `snakemake-interface-storage-plugins`.","message":"Snakemake's storage plugin interface versioning requires the plugin to be compatible. If Snakemake is updated to a new major interface version, the S3 plugin might need an update as well.","severity":"gotcha","affected_versions":"Any version of the plugin that is not aligned with the Snakemake core's storage interface version (e.g., plugin 0.3.2 adapted to interface 4.0)."},{"fix":"Upgrade to version 0.3.4 or newer to fix issues with nested directory retrieval and `FileExists` errors when getting directories.","message":"In earlier versions, nested directories were not always retrieved correctly, leading to missing files or `FileNotFoundError`.","severity":"breaking","affected_versions":"Versions prior to 0.2.12 (for nested directories) and 0.3.4 (for `FileExists` bug)."}],"env_vars":null,"last_verified":"2026-04-16T00:00:00.000Z","next_check":"2026-07-15T00:00:00.000Z","problems":[{"fix":"Ensure `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables are set, or configure `~/.aws/credentials` correctly. For MinIO, also ensure `AWS_ENDPOINT_URL` is set if connecting to a custom endpoint.","cause":"The plugin (via `boto3`) could not find valid AWS credentials in environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY), shared credential files (`~/.aws/credentials`), or IAM roles.","error":"botocore.exceptions.NoCredentialsError: Unable to locate credentials"},{"fix":"Set the `AWS_REGION` environment variable to the correct region for your S3 bucket (e.g., `us-east-1`). Ensure the region is consistent across your configuration.","cause":"The AWS region specified (either in environment variables or configuration) does not match the region where the bucket is being created or is expected to exist.","error":"botocore.errorfactory.ClientError: An error occurred (InvalidLocationConstraint) when calling the CreateBucket operation: The specified location constraint is not valid"},{"fix":"Double-check your `AWS_ENDPOINT_URL` (if used), `AWS_ACCESS_KEY_ID`, and `AWS_SECRET_ACCESS_KEY`. Ensure network connectivity to the S3 service. Upgrade `boto3` and the storage plugin to the latest versions.","cause":"This error can occur if `boto3` or underlying S3 operations fail to initialize an S3 client correctly, often due to misconfiguration or transient network issues, especially for non-AWS S3 endpoints.","error":"AttributeError: 'NoneType' object has no attribute 'meta'"},{"fix":"Verify that your S3 credentials have the necessary read/write permissions for the specified bucket and path. Check that the S3 URI (`s3://bucket/path/to/file`) is correct and the object actually exists.","cause":"Snakemake reports a `StorageException` when the underlying S3 operation fails, which could be due to incorrect permissions on the bucket/object, the object not existing, or other S3-specific errors.","error":"snakemake.exceptions.StorageException: Failed to retrieve file from S3: ... (e.g., Access Denied, NoSuchKey)"}]}