S3TorchConnectorClient

Warnings

• breaking In version 1.5.0, the internal S3Client now returns `HeadObjectResult` instead of `ObjectInfo`. `HeadObjectResult` does not include the `key` field, which might break custom reader implementations that directly relied on the `key` field from `ObjectInfo`.
  Fix: Update custom S3 client logic to use `HeadObjectResult` and adapt if the `key` field was previously accessed. For most users leveraging `s3torchconnector`'s high-level APIs, this change is internal and should not require direct modification.
• breaking Starting with version 1.5.0, `DCPOptimizedS3Reader` became the new default reader for `S3StorageReader` in `s3torchconnector`. While this offers performance improvements, it might lead to behavioral changes, especially with specific access patterns or error handling.
  Fix: Review the documentation for `DCPOptimizedS3Reader Errors` and test existing workloads thoroughly. If previous behavior is required, explicitly configure `S3StorageReader` with `S3ReaderConstructor.sequential()` or `S3ReaderConstructor.range_based()`.
• deprecated Python 3.8 support is being deprecated and will be removed in a future release. PyTorch itself has stopped supporting Python 3.8 after v2.4.1.
  Fix: Upgrade to Python 3.9 or newer. The library currently supports Python 3.8-3.14.
• deprecated macOS x86_64 wheel support will be deprecated in a future release.
  Fix: Users on macOS x86_64 should plan to migrate to an ARM-based Mac or use a Linux environment.
• gotcha Beginning with `boto3` v1.36.0, AWS SDK for Python introduced new default integrity protections for S3 clients (checksums on Put and validation on Get). This can cause issues when interacting with third-party S3-compatible services that may not fully support these new defaults.
  Fix: If experiencing issues with third-party S3 services, you can disable the new integrity protections by setting `request_checksum_calculation='when_required'` and `response_checksum_validation='when_required'` in your AWS configuration (e.g., via `~/.aws/config` or environment variables). It is generally not recommended to disable these for Amazon S3 itself.
• gotcha `S3Reader` instances (which are part of the `s3torchconnector`'s underlying read mechanism, utilizing `s3torchconnectorclient`) are not thread-safe.
  Fix: Do not share `S3Reader` instances across multiple threads. For multiprocessing with `DataLoader`, each worker process automatically creates its own `S3Reader` instance, which is the recommended pattern.

Install

• pip install s3torchconnectorclient Install stable version

Imports

• S3ClientConfig

  from s3torchconnectorclient import S3ClientConfig

  Used for fine-grained configuration of the underlying S3 client behavior. Typically passed to constructors in the `s3torchconnector` library.

Quickstart

This quickstart demonstrates how to directly configure the `S3ClientConfig` which is part of `s3torchconnectorclient`. While direct interaction with this low-level client is possible, it's more commonly used by passing `S3ClientConfig` instances to higher-level APIs provided by the `s3torchconnector` library for datasets and readers. This example sets a custom part size and throughput target. Ensure AWS credentials are configured (e.g., via environment variables or AWS CLI) for actual S3 operations.

import os
from s3torchconnectorclient import S3ClientConfig

# Configure the S3 client directly. This configuration is typically
# passed to higher-level constructors within the `s3torchconnector` library.
config = S3ClientConfig(
    part_size=10 * 1024 * 1024, # Example: 10 MiB part size for transfers
    throughput_target_gbps=5.0, # Example: Target 5 Gbps throughput
    profile=os.environ.get('AWS_PROFILE', None) # Use an AWS profile if specified
)

print(f"S3ClientConfig created with part size: {config.part_size / (1024*1024):.1f} MiB")
print(f"S3ClientConfig created with throughput target: {config.throughput_target_gbps} Gbps")
print(f"S3ClientConfig using AWS profile: {config.profile}")

# In a real application, 'config' would typically be used like this (requires 's3torchconnector'):
# from s3torchconnector import S3ReaderConstructor
# reader_constructor = S3ReaderConstructor.sequential(s3_client_config=config)
# dataset = S3MapDataset.from_prefix(DATASET_URI, region=REGION, reader_constructor=reader_constructor)