Snakemake S3 Storage Plugin

0.3.6 · active · verified Thu Apr 16

The `snakemake-storage-plugin-s3` library provides a Snakemake storage plugin for interacting with S3-compatible object storage systems, including AWS S3, MinIO, and others. It allows Snakemake workflows to read and write files directly from/to S3 buckets using `s3://` URIs. The project is actively maintained with frequent bug fix releases, typically every few months, ensuring compatibility and robustness.

Common errors

```
botocore.exceptions.NoCredentialsError: Unable to locate credentials
```
cause The plugin (via `boto3`) could not find valid AWS credentials in environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY), shared credential files (`~/.aws/credentials`), or IAM roles.

fix Ensure `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables are set, or configure `~/.aws/credentials` correctly. For MinIO, also ensure `AWS_ENDPOINT_URL` is set if connecting to a custom endpoint.
```
botocore.errorfactory.ClientError: An error occurred (InvalidLocationConstraint) when calling the CreateBucket operation: The specified location constraint is not valid
```
cause The AWS region specified (either in environment variables or configuration) does not match the region where the bucket is being created or is expected to exist.

fix Set the `AWS_REGION` environment variable to the correct region for your S3 bucket (e.g., `us-east-1`). Ensure the region is consistent across your configuration.
```
AttributeError: 'NoneType' object has no attribute 'meta'
```
cause This error can occur if `boto3` or underlying S3 operations fail to initialize an S3 client correctly, often due to misconfiguration or transient network issues, especially for non-AWS S3 endpoints.

fix Double-check your `AWS_ENDPOINT_URL` (if used), `AWS_ACCESS_KEY_ID`, and `AWS_SECRET_ACCESS_KEY`. Ensure network connectivity to the S3 service. Upgrade `boto3` and the storage plugin to the latest versions.
```
snakemake.exceptions.StorageException: Failed to retrieve file from S3: ... (e.g., Access Denied, NoSuchKey)
```
cause Snakemake reports a `StorageException` when the underlying S3 operation fails, which could be due to incorrect permissions on the bucket/object, the object not existing, or other S3-specific errors.

fix Verify that your S3 credentials have the necessary read/write permissions for the specified bucket and path. Check that the S3 URI (`s3://bucket/path/to/file`) is correct and the object actually exists.

Warnings

gotcha S3 object keys cannot contain `..` (double-dot) for path traversal. The plugin attempts to normalize paths, but directly using `..` in S3 URIs will lead to errors.
Fix: Ensure all S3 object paths used in your Snakefile are absolute or relative paths without `..` segments. For example, use `s3://bucket/dir/file.txt` instead of `s3://bucket/dir/../file.txt`.
gotcha When using non-AWS S3 compatible services (e.g., MinIO), `boto3` might automatically attach checksums to uploaded files which can cause issues or unexpected behavior.
Fix: Upgrade to version 0.3.3 or newer, which includes a workaround for this `boto3` behavior. If issues persist, consult `boto3` documentation or your S3 provider for specific configuration requirements regarding checksums.
gotcha Snakemake's storage plugin interface versioning requires the plugin to be compatible. If Snakemake is updated to a new major interface version, the S3 plugin might need an update as well.
Fix: Always keep the `snakemake-storage-plugin-s3` library updated to its latest version to ensure compatibility with recent Snakemake releases and the `snakemake-interface-storage-plugins`.
breaking In earlier versions, nested directories were not always retrieved correctly, leading to missing files or `FileNotFoundError`.
Fix: Upgrade to version 0.3.4 or newer to fix issues with nested directory retrieval and `FileExists` errors when getting directories.

Install

pip install snakemake-storage-plugin-s3 Install plugin

Imports

S3Storage
```
This plugin is automatically discovered by Snakemake when installed, and is used via 's3://' URIs in your Snakefile, not by direct Python import of a specific class for usage.
```
You do not typically import classes directly from this library into your Python code. Snakemake discovers and utilizes the plugin automatically when it's installed in the same environment.

Quickstart

This quickstart demonstrates how to use the S3 storage plugin in a `Snakefile`. Snakemake automatically recognizes `s3://` prefixes for input and output files when the plugin is installed. Ensure your S3 credentials are configured in your environment or via standard `boto3` configuration methods (e.g., `~/.aws/credentials`). For S3-compatible services like MinIO, `AWS_ENDPOINT_URL` is typically required.

# file: Snakefile

# Set S3 credentials via environment variables for a runnable example
# export AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY"
# export AWS_SECRET_ACCESS_KEY="YOUR_SECRET_KEY"
# export AWS_ENDPOINT_URL="http://localhost:9000" # Optional, for MinIO or other S3-compatible storages

rule all:
    input: "s3://my-test-bucket/output.txt"

rule generate_s3_file:
    output: "s3://my-test-bucket/output.txt"
    shell:
        "echo 'Hello Snakemake S3!' > {output}"

# To run this example:
# 1. Ensure snakemake and snakemake-storage-plugin-s3 are installed.
# 2. Set your S3 credentials (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION, etc.)
#    or AWS_ENDPOINT_URL for compatible S3 services.
# 3. Create 'my-test-bucket' in your S3 service.
# 4. Navigate to the directory containing this Snakefile and run: snakemake

view raw JSON →