Type Annotations for s3transfer

raw JSON →
0.16.0 verified Tue May 12 auth: no python install: stale quickstart: stale

types-s3transfer provides static type annotations and code completion for the `s3transfer` package, an Amazon S3 Transfer Manager. It significantly enhances IDE support and enables robust static type checking for Python applications that utilize `s3transfer` for efficient file transfers to and from AWS S3. Part of the `mypy_boto3_builder` project, this package is actively maintained, with its version (0.16.0) intended to align with the `s3transfer` version it types.

pip install types-s3transfer
error Missing type stubs for package 's3transfer'
cause The mypy type checker cannot find type information for the `s3transfer` library because the `types-s3transfer` stub package is not installed or configured correctly.
fix
Install the types-s3transfer package to provide the necessary type annotations: pip install types-s3transfer
error ModuleNotFoundError: No module named 'types_s3transfer'
cause The `types-s3transfer` package provides only type stubs and is not meant to be imported directly; the actual runtime code resides in the `s3transfer` package.
fix
Import s3transfer instead, ensuring both s3transfer and types-s3transfer are installed for runtime functionality and type checking, respectively: from s3transfer.manager import TransferManager
error ModuleNotFoundError: No module named 's3transfer'
cause The core `s3transfer` library, which contains the actual runtime code, has not been installed.
fix
Install the s3transfer package. For full type checking support, also install types-s3transfer: pip install s3transfer types-s3transfer
error AttributeError: module 's3transfer' has no attribute 'S3Transfer'
cause The `s3transfer` library's primary interface for transfer operations is `TransferManager`, not `S3Transfer`, or the module structure has changed in newer versions.
fix
Use TransferManager to manage S3 transfers. Ensure s3transfer and types-s3transfer are installed for correct runtime behavior and static type checking: from s3transfer.manager import TransferManager
gotcha It's crucial to install `s3transfer` alongside `types-s3transfer`. `types-s3transfer` provides only type stubs and does not contain the runtime implementation. Without `s3transfer` installed, your code will fail at runtime with `ModuleNotFoundError`.
fix Ensure both `s3transfer` and `types-s3transfer` are installed: `pip install s3transfer types-s3transfer`.
gotcha Ensure the version of `types-s3transfer` installed closely matches the version of `s3transfer` you are using. Mismatched versions can lead to incorrect type hints or false positive type errors, as the API might have changed between `s3transfer` versions but the stubs don't reflect that change.
fix Consult the `types-s3transfer` release notes or `s3transfer` documentation to determine compatible versions. Ideally, keep them in sync, e.g., `pip install 's3transfer==X.Y.Z' 'types-s3transfer==X.Y.Z'`.
breaking The return type of `TransferFuture.result()` was fixed in `types-s3transfer` version 0.6.0.post2 (and likely corresponding `s3transfer` versions) to correctly return `str`. Older versions of the stubs might have had incorrect `Any` types, potentially masking real type issues or causing confusion.
fix Upgrade to `types-s3transfer` version `0.6.0.post2` or newer to get the correct `str` return type for `TransferFuture.result()`: `pip install --upgrade types-s3transfer`.
gotcha Some users have reported `ImportError` for components like `RetriesExceededError` from `s3transfer.exceptions`, often due to version conflicts between `boto3`, `botocore`, and `s3transfer`. While `types-s3transfer` itself is stubs, these underlying runtime issues can affect how correctly the stubs are applied or whether the application even runs.
fix Ensure a compatible set of `boto3`, `botocore`, and `s3transfer` versions. Often, `pip install boto3` will install compatible `botocore` and `s3transfer` versions. If issues persist, try `pip install 's3transfer==0.4.0'` as a known stable combination with certain `boto3` versions, or refer to `boto3`'s dependency matrix.
breaking `boto3` is a required dependency for using `s3transfer` functionality with AWS S3. If `boto3` is not installed, you will encounter a `ModuleNotFoundError` when attempting to import it, preventing any AWS S3 operations.
fix Install `boto3`: `pip install boto3`.
breaking The package `boto3` is a common dependency when working with AWS services, including those that `s3transfer` interacts with. If `boto3` is not installed, any attempt to import it will result in a `ModuleNotFoundError`.
fix Ensure `boto3` is installed in your environment: `pip install boto3`.
python os / libc status wheel install import disk
3.10 alpine (musl) wheel - - 17.9M
3.10 alpine (musl) - - - -
3.10 slim (glibc) wheel 1.5s - 18M
3.10 slim (glibc) - - - -
3.11 alpine (musl) wheel - - 19.7M
3.11 alpine (musl) - - - -
3.11 slim (glibc) wheel 1.6s - 20M
3.11 slim (glibc) - - - -
3.12 alpine (musl) wheel - - 11.6M
3.12 alpine (musl) - - - -
3.12 slim (glibc) wheel 1.5s - 12M
3.12 slim (glibc) - - - -
3.13 alpine (musl) wheel - - 11.3M
3.13 alpine (musl) - - - -
3.13 slim (glibc) wheel 1.4s - 12M
3.13 slim (glibc) - - - -
3.9 alpine (musl) wheel - - 17.4M
3.9 alpine (musl) - - - -
3.9 slim (glibc) wheel 1.7s - 18M
3.9 slim (glibc) - - - -

This quickstart demonstrates how to use the underlying `s3transfer` library for uploading a file to S3. With `types-s3transfer` installed in your environment, your IDE and static type checker (e.g., MyPy, Pyright) will provide accurate type hints for `S3Transfer`, `TransferConfig`, and `TransferFuture` objects, improving development experience and helping catch type-related errors before runtime.

import boto3
from s3transfer.manager import S3Transfer, TransferConfig
from s3transfer.futures import TransferFuture
import os

def upload_file_to_s3(local_filepath: str, bucket_name: str, s3_key: str) -> TransferFuture:
    # Initialize S3 client
    s3_client = boto3.client(
        's3',
        region_name=os.environ.get('AWS_REGION', 'us-east-1'),
        aws_access_key_id=os.environ.get('AWS_ACCESS_KEY_ID', 'DUMMY_KEY'),
        aws_secret_access_key=os.environ.get('AWS_SECRET_ACCESS_KEY', 'DUMMY_SECRET')
    )

    # Configure transfer (optional)
    transfer_config = TransferConfig(
        multipart_threshold=8 * 1024 * 1024, # 8 MB
        max_concurrency=10
    )

    # Create S3Transfer manager
    transfer_manager = S3Transfer(s3_client, transfer_config)

    # Perform upload
    future: TransferFuture = transfer_manager.upload_file(local_filepath, bucket_name, s3_key)
    print(f"Uploading {local_filepath} to s3://{bucket_name}/{s3_key}")
    return future

if __name__ == '__main__':
    # Example usage (replace with actual values for testing)
    dummy_filepath = 'temp_file.txt'
    dummy_bucket = 'your-test-bucket'
    dummy_s3_key = 'test-upload/my_document.txt'

    # Create a dummy file
    with open(dummy_filepath, 'w') as f:
        f.write('Hello from types-s3transfer quickstart!')

    print(f"Created dummy file: {dummy_filepath}")

    try:
        # The 'types-s3transfer' package ensures that 'future' is correctly typed as TransferFuture
        future = upload_file_to_s3(dummy_filepath, dummy_bucket, dummy_s3_key)
        future.result() # Wait for the upload to complete
        print(f"Upload successful: s3://{dummy_bucket}/{dummy_s3_key}")
    except Exception as e:
        print(f"Upload failed: {e}")
    finally:
        # Clean up the dummy file
        if os.path.exists(dummy_filepath):
            os.remove(dummy_filepath)
            print(f"Removed dummy file: {dummy_filepath}")

# To benefit from type hints, ensure 'types-s3transfer' is installed in your environment.
# Type checkers like MyPy or Pyright will then provide suggestions and error checking.