s3transfer

Warnings

• breaking s3transfer is explicitly NOT GA. Interfaces can break between minor versions (e.g. 0.x.0 → 0.y.0). Always pin to a minor version in production requirements (e.g. s3transfer>=0.16.0,<0.17.0).
  Fix: Pin: s3transfer>=0.16.0,<0.17.0. For a more stable API surface, use the methods injected into the boto3 S3 client (s3_client.upload_file, s3_client.download_file) which are documented as stable.
• breaking Mismatched awscrt version causes ImportError at import time. If awscrt is installed (e.g. pulled in by boto3[crt]) but is not at the version expected by the installed s3transfer/boto3, 'from boto3.s3.transfer import TransferConfig' raises ImportError: cannot import name 'S3ResponseError' from 'awscrt.s3'.
  Fix: Upgrade all three in lockstep: pip install --upgrade boto3 botocore s3transfer. If awscrt must stay pinned, set preferred_transfer_client='classic' or uninstall awscrt entirely.
• breaking TransferConfig parameter names differ between s3transfer.manager.TransferConfig and boto3.s3.transfer.TransferConfig. The boto3 version maps max_concurrency→max_request_concurrency and max_io_queue→max_io_queue_size. Passing the raw s3transfer config to boto3 code (or vice versa) silently ignores aliased parameters.
  Fix: Always import TransferConfig from boto3.s3.transfer when working with the boto3 S3Transfer class.
• gotcha S3Transfer requires either a boto3 client OR a TransferManager instance — not both, not neither. Passing manager together with client, config, or osutil raises ValueError. Passing nothing also raises ValueError.
  Fix: Use S3Transfer(client=my_boto3_client) or S3Transfer(manager=my_manager), never mixing both argument groups.
• gotcha CRC32 is now the default checksum algorithm for uploads. If the receiving side or downstream tooling does not handle the x-amz-checksum-crc32 header (e.g., some S3-compatible stores), uploads may fail or produce unexpected validation errors.
  Fix: To disable automatic checksums, ensure botocore config request_checksum_calculation is set to 'when_required', or explicitly pass ChecksumAlgorithm in extra_args.
• gotcha Thread-local context (AWS X-Ray segments, OpenTelemetry spans, contextvars) is lost inside transfer worker threads. The transfer manager uses a ThreadPoolExecutor internally; tracing SDKs that rely on thread-local or context-variable storage will not see the parent trace context in worker threads.
  Fix: Use TransferConfig(use_threads=False) for single-threaded execution (loses concurrency) or propagate context manually via subscriber callbacks.
• gotcha download_file and upload_file require filename to be a str or os.PathLike. Passing a raw bytes path or any other object raises ValueError immediately without touching S3.
  Fix: Convert pathlib.Path objects (accepted via os.fspath internally) or ensure the filename argument is a str. Example: transfer.download_file(bucket, key, str(path_obj)).

Install

• pip install s3transfer Core (no CRT)
• pip install "s3transfer" "boto3[crt]" With AWS CRT acceleration (via boto3 extra)

Imports

• TransferManager

  from s3transfer.manager import TransferManager

  Low-level public API. Prefer boto3 client injection (s3_client.upload_file / download_file) for the most stable surface.
• TransferConfig

  from boto3.s3.transfer import TransferConfig

  boto3's TransferConfig adds aliases (max_concurrency, max_io_queue) and preferred_transfer_client on top of s3transfer's internal TransferConfig. Using the raw s3transfer one skips those aliases and the CRT dispatch logic.
• S3Transfer

  from boto3.s3.transfer import S3Transfer

  S3Transfer lives in boto3.s3.transfer, not directly on the s3transfer top-level namespace. Only classes explicitly documented in the boto3 S3 customization reference are considered public stable API.
• BaseSubscriber

  from s3transfer.subscribers import BaseSubscriber

  Subclass this to implement on_queued, on_progress, and on_done progress callbacks.
• RetriesExceededError

  from boto3.exceptions import RetriesExceededError

  boto3 wraps s3transfer's RetriesExceededError in its own exception for backwards compatibility. Catching the s3transfer version will miss errors raised through the boto3 S3Transfer shim.

Quickstart

Upload and download a file using TransferManager via the stable boto3.s3.transfer interface, with a custom TransferConfig.

import os
import boto3
from boto3.s3.transfer import TransferConfig, S3Transfer

# Credentials resolved from env, ~/.aws/credentials, or IAM role
aws_access_key = os.environ.get('AWS_ACCESS_KEY_ID', '')
aws_secret_key = os.environ.get('AWS_SECRET_ACCESS_KEY', '')
region = os.environ.get('AWS_DEFAULT_REGION', 'us-east-1')
bucket = os.environ.get('S3_BUCKET', 'my-bucket')

client = boto3.client(
    's3',
    region_name=region,
    aws_access_key_id=aws_access_key or None,
    aws_secret_access_key=aws_secret_key or None,
)

config = TransferConfig(
    multipart_threshold=8 * 1024 * 1024,   # 8 MB
    max_concurrency=10,
    num_download_attempts=5,
    use_threads=True,
    # Force classic manager; use 'auto' to allow CRT when awscrt is installed
    preferred_transfer_client='classic',
)

transfer = S3Transfer(client, config)

# Upload
transfer.upload_file(
    '/tmp/example.txt',
    bucket,
    'uploads/example.txt',
    extra_args={'ContentType': 'text/plain'},
)

# Download
transfer.download_file(
    bucket,
    'uploads/example.txt',
    '/tmp/example_downloaded.txt',
)

print('Transfer complete')