s3transfer

0.16.0 verified Tue May 12 auth: no python install: verified quickstart: stale

s3transfer is an Amazon S3 Transfer Manager for Python, maintained by Amazon Web Services. It provides managed multipart uploads and downloads, automatic retries, progress callbacks, bandwidth throttling, and optional AWS CRT-accelerated transfers. As of version 0.16.0 it defaults to CRC32 checksums and supports user-provided full-object checksums. The project is NOT yet generally available (GA); minor-version bumps may contain breaking interface changes, so production deployments must pin to a minor version. It is bundled as a dependency of boto3 and botocore and is typically consumed indirectly through those SDKs.

pip install s3transfer

Common errors

error ModuleNotFoundError: No module named 's3transfer' ↓

cause The `s3transfer` package is not installed in the current Python environment or is not accessible, even though it's typically a dependency of `boto3`.

fix

Install boto3 (which includes s3transfer as a dependency) or explicitly install s3transfer. pip install boto3 # Or, if only s3transfer is needed: pip install s3transfer

error A client error (InvalidRequest) occurred when calling the PutObject operation: Content-MD5 HTTP header is inconsistent, please remove and try again. ↓

cause When using `boto3`'s high-level S3 transfer methods (`upload_file`, `upload_fileobj`) which utilize `s3transfer`, the library or S3 service handles checksums automatically; explicitly providing a `ContentMD5` header in `ExtraArgs` can conflict.

fix

Avoid manually providing the ContentMD5 header in ExtraArgs when using boto3's high-level S3 transfer methods. s3_client.upload_file( 'local_file.txt', 'my-bucket', 'remote_file.txt' # Remove ExtraArgs={'ContentMD5': 'your_md5_hash_base64'} )

error TypeError: __init__() got an unexpected keyword argument 'some_invalid_argument' ↓

cause The `boto3.s3.transfer.TransferConfig` class (part of `s3transfer`'s interface via `boto3`) was initialized with an argument that does not exist or is misspelled for the installed version of `boto3`/`s3transfer`.

fix

Review the official boto3 documentation for the TransferConfig class to ensure all arguments used are valid for your installed version, correcting any typos or removing deprecated arguments. from boto3.s3.transfer import TransferConfig # Example with valid arguments config = TransferConfig( multipart_threshold=1024 * 25, # 25 MB max_concurrency=10, use_threads=True ) # s3_client.upload_file('local_file.txt', 'my-bucket', 'remote_file.txt', Config=config)

error ModuleNotFoundError: No module named 'awscrt' ↓

cause AWS CRT-accelerated transfers, an optional feature, require the `awscrt` Python package to be installed in the environment.

fix

Install the awscrt package in your Python environment to enable CRT acceleration. pip install awscrt

Warnings

breaking s3transfer is explicitly NOT GA. Interfaces can break between minor versions (e.g. 0.x.0 → 0.y.0). Always pin to a minor version in production requirements (e.g. s3transfer>=0.16.0,<0.17.0). ↓

fix Pin: s3transfer>=0.16.0,<0.17.0. For a more stable API surface, use the methods injected into the boto3 S3 client (s3_client.upload_file, s3_client.download_file) which are documented as stable.

breaking Mismatched awscrt version causes ImportError at import time. If awscrt is installed (e.g. pulled in by boto3[crt]) but is not at the version expected by the installed s3transfer/boto3, 'from boto3.s3.transfer import TransferConfig' raises ImportError: cannot import name 'S3ResponseError' from 'awscrt.s3'. ↓

fix Upgrade all three in lockstep: pip install --upgrade boto3 botocore s3transfer. If awscrt must stay pinned, set preferred_transfer_client='classic' or uninstall awscrt entirely.

breaking TransferConfig parameter names differ between s3transfer.manager.TransferConfig and boto3.s3.transfer.TransferConfig. The boto3 version maps max_concurrency→max_request_concurrency and max_io_queue→max_io_queue_size. Passing the raw s3transfer config to boto3 code (or vice versa) silently ignores aliased parameters. ↓

fix Always import TransferConfig from boto3.s3.transfer when working with the boto3 S3Transfer class.

gotcha S3Transfer requires either a boto3 client OR a TransferManager instance — not both, not neither. Passing manager together with client, config, or osutil raises ValueError. Passing nothing also raises ValueError. ↓

fix Use S3Transfer(client=my_boto3_client) or S3Transfer(manager=my_manager), never mixing both argument groups.

gotcha CRC32 is now the default checksum algorithm for uploads. If the receiving side or downstream tooling does not handle the x-amz-checksum-crc32 header (e.g., some S3-compatible stores), uploads may fail or produce unexpected validation errors. ↓

fix To disable automatic checksums, ensure botocore config request_checksum_calculation is set to 'when_required', or explicitly pass ChecksumAlgorithm in extra_args.

gotcha Thread-local context (AWS X-Ray segments, OpenTelemetry spans, contextvars) is lost inside transfer worker threads. The transfer manager uses a ThreadPoolExecutor internally; tracing SDKs that rely on thread-local or context-variable storage will not see the parent trace context in worker threads. ↓

fix Use TransferConfig(use_threads=False) for single-threaded execution (loses concurrency) or propagate context manually via subscriber callbacks.

gotcha download_file and upload_file require filename to be a str or os.PathLike. Passing a raw bytes path or any other object raises ValueError immediately without touching S3. ↓

fix Convert pathlib.Path objects (accepted via os.fspath internally) or ensure the filename argument is a str. Example: transfer.download_file(bucket, key, str(path_obj)).

breaking When using `upload_file` or `download_file`, a `FileNotFoundError` will be raised if the specified local file (for upload) or the directory path (for download) does not exist. ↓

fix Ensure the local file exists at the specified `filename` path before calling `upload_file`. For `download_file`, ensure that parent directories for the target `filename` path exist before the operation.

breaking The local file specified for upload_file or download_file does not exist. This results in a FileNotFoundError when the transfer manager attempts to access the file's metadata (e.g., size) or content. ↓

fix Ensure the source file exists on the local filesystem before calling upload_file. For download_file, ensure the destination directory is writable and accessible.

Install

pip install "s3transfer" "boto3[crt]"

Install compatibility verified last tested: 2026-05-12

python os / libc variant status wheel install import disk

3.10 alpine (musl) "s3transfer" - - 0.54s 61.8M

3.10 alpine (musl) s3transfer - - 0.49s 48.8M

3.10 slim (glibc) "s3transfer" - - 0.40s 63M

3.10 slim (glibc) s3transfer - - 0.38s 49M

3.11 alpine (musl) "s3transfer" - - 0.78s 64.7M

3.11 alpine (musl) s3transfer - - 0.70s 51.6M

3.11 slim (glibc) "s3transfer" - - 0.66s 66M

3.11 slim (glibc) s3transfer - - 0.61s 52M

3.12 alpine (musl) "s3transfer" - - 0.67s 56.3M

3.12 alpine (musl) s3transfer - - 0.61s 43.3M

3.12 slim (glibc) "s3transfer" - - 0.66s 57M

3.12 slim (glibc) s3transfer - - 0.58s 44M

3.13 alpine (musl) "s3transfer" - - 0.66s 55.9M

3.13 alpine (musl) s3transfer - - 0.59s 42.9M

3.13 slim (glibc) "s3transfer" - - 0.64s 57M

3.13 slim (glibc) s3transfer - - 0.60s 43M

3.9 alpine (musl) "s3transfer" - - 0.46s 61.3M

3.9 alpine (musl) s3transfer - - 0.42s 48.3M

3.9 slim (glibc) "s3transfer" - - 0.41s 62M

3.9 slim (glibc) s3transfer - - 0.39s 49M

Imports

TransferManager
```
from s3transfer.manager import TransferManager
```
Low-level public API. Prefer boto3 client injection (s3_client.upload_file / download_file) for the most stable surface.
TransferConfig
wrong
```
from s3transfer.manager import TransferConfig
```
correct
```
from boto3.s3.transfer import TransferConfig
```
boto3's TransferConfig adds aliases (max_concurrency, max_io_queue) and preferred_transfer_client on top of s3transfer's internal TransferConfig. Using the raw s3transfer one skips those aliases and the CRT dispatch logic.
S3Transfer
wrong
```
import s3transfer; s3transfer.S3Transfer(...)
```
correct
```
from boto3.s3.transfer import S3Transfer
```
S3Transfer lives in boto3.s3.transfer, not directly on the s3transfer top-level namespace. Only classes explicitly documented in the boto3 S3 customization reference are considered public stable API.
BaseSubscriber
```
from s3transfer.subscribers import BaseSubscriber
```
Subclass this to implement on_queued, on_progress, and on_done progress callbacks.
RetriesExceededError
wrong
```
from s3transfer.exceptions import RetriesExceededError
```
correct
```
from boto3.exceptions import RetriesExceededError
```
boto3 wraps s3transfer's RetriesExceededError in its own exception for backwards compatibility. Catching the s3transfer version will miss errors raised through the boto3 S3Transfer shim.

Quickstart stale last tested: 2026-04-23

Upload and download a file using TransferManager via the stable boto3.s3.transfer interface, with a custom TransferConfig.

import os
import boto3
from boto3.s3.transfer import TransferConfig, S3Transfer

# Credentials resolved from env, ~/.aws/credentials, or IAM role
aws_access_key = os.environ.get('AWS_ACCESS_KEY_ID', '')
aws_secret_key = os.environ.get('AWS_SECRET_ACCESS_KEY', '')
region = os.environ.get('AWS_DEFAULT_REGION', 'us-east-1')
bucket = os.environ.get('S3_BUCKET', 'my-bucket')

client = boto3.client(
    's3',
    region_name=region,
    aws_access_key_id=aws_access_key or None,
    aws_secret_access_key=aws_secret_key or None,
)

config = TransferConfig(
    multipart_threshold=8 * 1024 * 1024,   # 8 MB
    max_concurrency=10,
    num_download_attempts=5,
    use_threads=True,
    # Force classic manager; use 'auto' to allow CRT when awscrt is installed
    preferred_transfer_client='classic',
)

transfer = S3Transfer(client, config)

# Upload
transfer.upload_file(
    '/tmp/example.txt',
    bucket,
    'uploads/example.txt',
    extra_args={'ContentType': 'text/plain'},
)

# Download
transfer.download_file(
    bucket,
    'uploads/example.txt',
    '/tmp/example_downloaded.txt',
)

print('Transfer complete')