{"id":270,"library":"s3transfer","title":"s3transfer","description":"s3transfer is an Amazon S3 Transfer Manager for Python, maintained by Amazon Web Services. It provides managed multipart uploads and downloads, automatic retries, progress callbacks, bandwidth throttling, and optional AWS CRT-accelerated transfers. As of version 0.16.0 it defaults to CRC32 checksums and supports user-provided full-object checksums. The project is NOT yet generally available (GA); minor-version bumps may contain breaking interface changes, so production deployments must pin to a minor version. It is bundled as a dependency of boto3 and botocore and is typically consumed indirectly through those SDKs.","status":"active","version":"0.16.0","language":"python","source_language":"en","source_url":"https://github.com/boto/s3transfer","tags":["aws","s3","transfer","upload","download","boto3","botocore","multipart","crt"],"install":[{"cmd":"pip install s3transfer","lang":"bash","label":"Core (no CRT)"},{"cmd":"pip install \"s3transfer\" \"boto3[crt]\"","lang":"bash","label":"With AWS CRT acceleration (via boto3 extra)"}],"dependencies":[{"reason":"Required core AWS SDK primitives; s3transfer requires botocore>=1.37.4,<2.0a.0","package":"botocore","optional":false},{"reason":"Optional CRT-based high-throughput transfer backend; auto-selected when installed and running on an optimized instance type","package":"awscrt","optional":true}],"imports":[{"note":"Low-level public API. Prefer boto3 client injection (s3_client.upload_file / download_file) for the most stable surface.","symbol":"TransferManager","correct":"from s3transfer.manager import TransferManager"},{"note":"boto3's TransferConfig adds aliases (max_concurrency, max_io_queue) and preferred_transfer_client on top of s3transfer's internal TransferConfig. Using the raw s3transfer one skips those aliases and the CRT dispatch logic.","wrong":"from s3transfer.manager import TransferConfig","symbol":"TransferConfig","correct":"from boto3.s3.transfer import TransferConfig"},{"note":"S3Transfer lives in boto3.s3.transfer, not directly on the s3transfer top-level namespace. Only classes explicitly documented in the boto3 S3 customization reference are considered public stable API.","wrong":"import s3transfer; s3transfer.S3Transfer(...)","symbol":"S3Transfer","correct":"from boto3.s3.transfer import S3Transfer"},{"note":"Subclass this to implement on_queued, on_progress, and on_done progress callbacks.","symbol":"BaseSubscriber","correct":"from s3transfer.subscribers import BaseSubscriber"},{"note":"boto3 wraps s3transfer's RetriesExceededError in its own exception for backwards compatibility. Catching the s3transfer version will miss errors raised through the boto3 S3Transfer shim.","wrong":"from s3transfer.exceptions import RetriesExceededError","symbol":"RetriesExceededError","correct":"from boto3.exceptions import RetriesExceededError"}],"quickstart":{"code":"import os\nimport boto3\nfrom boto3.s3.transfer import TransferConfig, S3Transfer\n\n# Credentials resolved from env, ~/.aws/credentials, or IAM role\naws_access_key = os.environ.get('AWS_ACCESS_KEY_ID', '')\naws_secret_key = os.environ.get('AWS_SECRET_ACCESS_KEY', '')\nregion = os.environ.get('AWS_DEFAULT_REGION', 'us-east-1')\nbucket = os.environ.get('S3_BUCKET', 'my-bucket')\n\nclient = boto3.client(\n    's3',\n    region_name=region,\n    aws_access_key_id=aws_access_key or None,\n    aws_secret_access_key=aws_secret_key or None,\n)\n\nconfig = TransferConfig(\n    multipart_threshold=8 * 1024 * 1024,   # 8 MB\n    max_concurrency=10,\n    num_download_attempts=5,\n    use_threads=True,\n    # Force classic manager; use 'auto' to allow CRT when awscrt is installed\n    preferred_transfer_client='classic',\n)\n\ntransfer = S3Transfer(client, config)\n\n# Upload\ntransfer.upload_file(\n    '/tmp/example.txt',\n    bucket,\n    'uploads/example.txt',\n    extra_args={'ContentType': 'text/plain'},\n)\n\n# Download\ntransfer.download_file(\n    bucket,\n    'uploads/example.txt',\n    '/tmp/example_downloaded.txt',\n)\n\nprint('Transfer complete')","lang":"python","description":"Upload and download a file using TransferManager via the stable boto3.s3.transfer interface, with a custom TransferConfig."},"warnings":[{"fix":"Pin: s3transfer>=0.16.0,<0.17.0. For a more stable API surface, use the methods injected into the boto3 S3 client (s3_client.upload_file, s3_client.download_file) which are documented as stable.","message":"s3transfer is explicitly NOT GA. Interfaces can break between minor versions (e.g. 0.x.0 → 0.y.0). Always pin to a minor version in production requirements (e.g. s3transfer>=0.16.0,<0.17.0).","severity":"breaking","affected_versions":"<1.0.0 (all current releases)"},{"fix":"Upgrade all three in lockstep: pip install --upgrade boto3 botocore s3transfer. If awscrt must stay pinned, set preferred_transfer_client='classic' or uninstall awscrt entirely.","message":"Mismatched awscrt version causes ImportError at import time. If awscrt is installed (e.g. pulled in by boto3[crt]) but is not at the version expected by the installed s3transfer/boto3, 'from boto3.s3.transfer import TransferConfig' raises ImportError: cannot import name 'S3ResponseError' from 'awscrt.s3'.","severity":"breaking","affected_versions":"Observed around boto3 1.33.x / s3transfer 0.8.x; can recur on any awscrt mismatch"},{"fix":"Always import TransferConfig from boto3.s3.transfer when working with the boto3 S3Transfer class.","message":"TransferConfig parameter names differ between s3transfer.manager.TransferConfig and boto3.s3.transfer.TransferConfig. The boto3 version maps max_concurrency→max_request_concurrency and max_io_queue→max_io_queue_size. Passing the raw s3transfer config to boto3 code (or vice versa) silently ignores aliased parameters.","severity":"breaking","affected_versions":"All versions"},{"fix":"Use S3Transfer(client=my_boto3_client) or S3Transfer(manager=my_manager), never mixing both argument groups.","message":"S3Transfer requires either a boto3 client OR a TransferManager instance — not both, not neither. Passing manager together with client, config, or osutil raises ValueError. Passing nothing also raises ValueError.","severity":"gotcha","affected_versions":"All versions"},{"fix":"To disable automatic checksums, ensure botocore config request_checksum_calculation is set to 'when_required', or explicitly pass ChecksumAlgorithm in extra_args.","message":"CRC32 is now the default checksum algorithm for uploads. If the receiving side or downstream tooling does not handle the x-amz-checksum-crc32 header (e.g., some S3-compatible stores), uploads may fail or produce unexpected validation errors.","severity":"gotcha","affected_versions":">=0.10.0 (CRC32 default introduced)"},{"fix":"Use TransferConfig(use_threads=False) for single-threaded execution (loses concurrency) or propagate context manually via subscriber callbacks.","message":"Thread-local context (AWS X-Ray segments, OpenTelemetry spans, contextvars) is lost inside transfer worker threads. The transfer manager uses a ThreadPoolExecutor internally; tracing SDKs that rely on thread-local or context-variable storage will not see the parent trace context in worker threads.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Convert pathlib.Path objects (accepted via os.fspath internally) or ensure the filename argument is a str. Example: transfer.download_file(bucket, key, str(path_obj)).","message":"download_file and upload_file require filename to be a str or os.PathLike. Passing a raw bytes path or any other object raises ValueError immediately without touching S3.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Ensure the local file exists at the specified `filename` path before calling `upload_file`. For `download_file`, ensure that parent directories for the target `filename` path exist before the operation.","message":"When using `upload_file` or `download_file`, a `FileNotFoundError` will be raised if the specified local file (for upload) or the directory path (for download) does not exist.","severity":"breaking","affected_versions":"All versions"},{"fix":"Ensure the source file exists on the local filesystem before calling upload_file. For download_file, ensure the destination directory is writable and accessible.","message":"The local file specified for upload_file or download_file does not exist. This results in a FileNotFoundError when the transfer manager attempts to access the file's metadata (e.g., size) or content.","severity":"breaking","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-05-12T12:40:33.770Z","next_check":"2026-06-26T00:00:00.000Z","problems":[{"fix":"Install `boto3` (which includes `s3transfer` as a dependency) or explicitly install `s3transfer`.\npip install boto3\n# Or, if only s3transfer is needed:\npip install s3transfer","cause":"The `s3transfer` package is not installed in the current Python environment or is not accessible, even though it's typically a dependency of `boto3`.","error":"ModuleNotFoundError: No module named 's3transfer'"},{"fix":"Avoid manually providing the `ContentMD5` header in `ExtraArgs` when using `boto3`'s high-level S3 transfer methods.\ns3_client.upload_file(\n    'local_file.txt', 'my-bucket', 'remote_file.txt'\n    # Remove ExtraArgs={'ContentMD5': 'your_md5_hash_base64'}\n)","cause":"When using `boto3`'s high-level S3 transfer methods (`upload_file`, `upload_fileobj`) which utilize `s3transfer`, the library or S3 service handles checksums automatically; explicitly providing a `ContentMD5` header in `ExtraArgs` can conflict.","error":"A client error (InvalidRequest) occurred when calling the PutObject operation: Content-MD5 HTTP header is inconsistent, please remove and try again."},{"fix":"Review the official `boto3` documentation for the `TransferConfig` class to ensure all arguments used are valid for your installed version, correcting any typos or removing deprecated arguments.\nfrom boto3.s3.transfer import TransferConfig\n\n# Example with valid arguments\nconfig = TransferConfig(\n    multipart_threshold=1024 * 25,  # 25 MB\n    max_concurrency=10,\n    use_threads=True\n)\n# s3_client.upload_file('local_file.txt', 'my-bucket', 'remote_file.txt', Config=config)","cause":"The `boto3.s3.transfer.TransferConfig` class (part of `s3transfer`'s interface via `boto3`) was initialized with an argument that does not exist or is misspelled for the installed version of `boto3`/`s3transfer`.","error":"TypeError: __init__() got an unexpected keyword argument 'some_invalid_argument'"},{"fix":"Install the `awscrt` package in your Python environment to enable CRT acceleration.\npip install awscrt","cause":"AWS CRT-accelerated transfers, an optional feature, require the `awscrt` Python package to be installed in the environment.","error":"ModuleNotFoundError: No module named 'awscrt'"}],"ecosystem":"pypi","meta_description":null,"install_score":100,"install_tag":"verified","quickstart_score":0,"quickstart_tag":"stale","pypi_latest":null,"install_checks":{"last_tested":"2026-05-12","tag":"verified","tag_description":"installs cleanly on critical runtimes, fast import, recently tested","results":[{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"crt","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.54,"mem_mb":11.6,"disk_size":"61.8M"},{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.49,"mem_mb":11.1,"disk_size":"48.8M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"crt","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.4,"mem_mb":11.6,"disk_size":"63M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.38,"mem_mb":11.1,"disk_size":"49M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"crt","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.78,"mem_mb":13.1,"disk_size":"64.7M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.7,"mem_mb":12.5,"disk_size":"51.6M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"crt","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.66,"mem_mb":13.1,"disk_size":"66M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.61,"mem_mb":12.5,"disk_size":"52M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"crt","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.67,"mem_mb":12.9,"disk_size":"56.3M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.61,"mem_mb":12.3,"disk_size":"43.3M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"crt","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.66,"mem_mb":12.8,"disk_size":"57M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.58,"mem_mb":12.3,"disk_size":"44M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"crt","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.66,"mem_mb":13.8,"disk_size":"55.9M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.59,"mem_mb":13.2,"disk_size":"42.9M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"crt","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.64,"mem_mb":13.8,"disk_size":"57M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.6,"mem_mb":13.2,"disk_size":"43M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"crt","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.46,"mem_mb":10.7,"disk_size":"61.3M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.42,"mem_mb":10.2,"disk_size":"48.3M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"crt","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.41,"mem_mb":10.7,"disk_size":"62M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.39,"mem_mb":10.2,"disk_size":"49M"}]},"quickstart_checks":{"last_tested":"2026-04-23","tag":"stale","tag_description":"widespread failures or data too old to trust","results":[{"runtime":"python:3.10-alpine","exit_code":1},{"runtime":"python:3.10-slim","exit_code":1},{"runtime":"python:3.11-alpine","exit_code":1},{"runtime":"python:3.11-slim","exit_code":1},{"runtime":"python:3.12-alpine","exit_code":1},{"runtime":"python:3.12-slim","exit_code":1},{"runtime":"python:3.13-alpine","exit_code":1},{"runtime":"python:3.13-slim","exit_code":1},{"runtime":"python:3.9-alpine","exit_code":1},{"runtime":"python:3.9-slim","exit_code":1}]}}